Nov 03 20:44:36.353716 ip-10-0-140-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 20:44:36.353725 ip-10-0-140-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 20:44:36.353732 ip-10-0-140-144 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 20:44:36.353949 ip-10-0-140-144 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 20:44:46.560037 ip-10-0-140-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 20:44:46.560052 ip-10-0-140-144 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f2de8a4b025447eaaa4475260c952258 --
Nov 03 20:47:05.768966 ip-10-0-140-144 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:47:06.056611 ip-10-0-140-144 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:47:06.056611 ip-10-0-140-144 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:47:06.056611 ip-10-0-140-144 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:47:06.056611 ip-10-0-140-144 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:47:06.061225 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061162    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.061225 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061225    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061230    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061233    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061237    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061240    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061243    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061246    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061250    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061253    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061256    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061258    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061261    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061264    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061267    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061270    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061272    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061275    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061278    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061282    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061287    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.061295 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061291    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061293    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061297    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061300    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061304    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061307    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061310    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061312    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061315    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061318    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061320    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061323    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061325    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061328    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061331    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061334    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061337    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061339    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061342    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.061775 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061345    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061348    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061350    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061353    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061355    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061358    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061360    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061363    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061365    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061368    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061370    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061373    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061375    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061378    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061381    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061384    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061387    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061391    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061394    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061396    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.062212 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061399    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061402    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061404    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061407    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061409    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061412    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061414    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061417    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061419    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061422    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061425    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061427    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061430    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061432    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061435    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061438    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061441    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061443    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061446    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061448    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.062699 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061451    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061453    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061458    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061462    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061465    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061468    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061471    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061474    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061477    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061479    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061482    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061484    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061487    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061489    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061492    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061495    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061497    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061501    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061503    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.063156 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061506    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061508    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061511    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061513    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061516    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061519    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.061522    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062300    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062308    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062313    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062316    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062320    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062324    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062326    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062329    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062332    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062335    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062337    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062340    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062343    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.063598 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062346    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062348    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062351    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062354    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062356    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062359    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062363    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062367    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062370    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062373    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062376    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062378    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062381    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062383    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062386    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062390    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062393    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062396    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062398    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.064086 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062401    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062404    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062407    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062410    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062413    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062416    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062419    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062421    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062424    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062426    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062429    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062432    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062435    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062438    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062441    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062443    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062446    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062449    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062452    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.064532 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062454    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062457    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062460    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062462    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062465    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062467    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062470    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062472    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062475    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062477    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062480    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062482    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062485    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062487    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062490    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062493    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062496    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062499    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062502    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062504    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.064991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062507    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062510    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062513    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062515    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062518    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062520    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062523    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062526    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062528    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062531    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062533    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062536    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062539    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062542    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062544    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062548    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062550    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062553    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062556    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.065459 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062559    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062562    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062564    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062567    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062580    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062583    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062586    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062588    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062592    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062595    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062597    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062600    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062603    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062606    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062609    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.062612    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063087    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063092    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063095    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.065917 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063098    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063101    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063103    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063106    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063109    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063111    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063114    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063117    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063120    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063123    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063125    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063128    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063131    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063134    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063137    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063139    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063142    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063145    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063148    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063151    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.066360 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063153    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063157    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063160    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063163    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063166    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063169    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063171    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063174    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063177    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063179    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063182    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063185    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063187    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063191    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063195    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063198    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063201    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063204    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063207    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063210    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.066841 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063215    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063217    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063220    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063223    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063225    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063228    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063231    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063234    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063236    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063239    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063242    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063244    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063247    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063250    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063253    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063256    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063258    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063261    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063263    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063266    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.067319 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063269    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063271    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063274    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063276    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063279    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063281    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063284    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063288    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063291    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063293    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063296    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063298    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063302    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063305    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063307    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063310    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063313    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063315    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063318    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.067788 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063321    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063323    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063326    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063328    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063331    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063333    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063336    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063339    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063341    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063344    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063348    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063352    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063355    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063358    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063360    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063363    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063366    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063368    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063371    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063373    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.068232 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063377    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063380    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063383    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.063386    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064614    2567 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064624    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064630    2567 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064645    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064649    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064653    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064657    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064661    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064664    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064667    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064671    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064674    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064677    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064680    2567 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064683    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064686    2567 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064688    2567 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064693    2567 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064697    2567 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:47:06.068741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064700    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064703    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064706    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064710    2567 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064713    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064717    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064720    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064724    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064727    2567 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064730    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064734    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064737    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064740    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064744    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064747    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064750    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064753    2567 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064756    2567 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064759    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064763    2567 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064766    2567 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064769    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064773    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064776    2567 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:47:06.069278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064780    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064783    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064787    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064790    2567 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064793    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064796    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064799    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064803    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064806    2567 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064809    2567 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064812    2567 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064817    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064820    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064823    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064827    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064830    2567 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064833    2567 flags.go:64] FLAG: --help="false"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064836    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064839    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064842    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064845    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064849    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064852    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:47:06.069892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064855    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064858    2567 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064861    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064864    2567 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064867    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064870    2567 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064873    2567 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064876    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064879    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064882    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064885    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064887    2567 flags.go:64] FLAG: --lock-file=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064890    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064893    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064897    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064902    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064905    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064908    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064912    2567 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064915    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064918    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064921    2567 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064924    2567 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064928    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064931    2567 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:47:06.070576 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064935    2567 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064938    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064941    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064944    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064947    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064950    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064953    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064956    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064963    2567 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064966    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064969    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064972    2567 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064975    2567 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064978    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064981    2567 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064983    2567 flags.go:64] FLAG: --port="10250"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064986    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064990    2567 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00aa1d9e4dc4bd4a9"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064993    2567 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064996    2567 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.064999    2567 flags.go:64] FLAG: --register-node="true"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065003    2567 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065007    2567 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065009    2567 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065012    2567 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065016    2567 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:47:06.071226 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065019    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065023    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065029    2567 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065032    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065035    2567 flags.go:64] FLAG: --runonce="false"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065038    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065041    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065044    2567 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065047    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065050    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065053    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065056    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065060    2567 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065063    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065066    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065068    2567 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065071    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065075    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065078    2567 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065081    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065086    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065089    2567 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065091    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065096    2567 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065099    2567 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:47:06.071934 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065101    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065104    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065107    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065110    2567 flags.go:64] FLAG: --v="2"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065114    2567 flags.go:64] FLAG: --version="false"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065118    2567 flags.go:64] FLAG: --vmodule=""
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065123    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065126    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065255    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065260    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065263    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065266    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065268    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065271    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065275    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065278    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065281    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065284    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065286    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065289    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065292    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.072650 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065294    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065297    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065300    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065302    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065305    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065308    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065310    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065313    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065315    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065320    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065324    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065327    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065330    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065333    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065337    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065341    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065344    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065347    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065350    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.073456 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065353    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065357    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065360    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065363    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065365    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065368    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065372    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065375    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065377    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065380    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065382    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065385    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065388    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065390    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065393    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065396    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065398    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065401    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065403    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065406    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.073930 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065409    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065411    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065414    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065417    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065419    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065422    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065425    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065427    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065430    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065433    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065435    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065438    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065441    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065445    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065448    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065450    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065453    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065456    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065460    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065463    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.074436 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065466    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065469    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065471    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065474    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065477    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065479    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065482    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065485    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065487    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065490    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065493    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065496    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065499    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065501    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065504    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065506    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065509    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065511    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065514    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065517    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.074935 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065519    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065522    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065525    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065527    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065530    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065534    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065537    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065539    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065542    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065545    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065548    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065551    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065554    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065557    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.065583    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065615    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.075405 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065620    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065624    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065628    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065631    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065647    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065651    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065655    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065660    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065663    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065665    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065668    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065671    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065673    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065676    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065678    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065681    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065684    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065686    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065689    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.075801 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065692    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065694    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065697    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065701    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065705    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065708    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065711    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065714    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065717    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065720    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065723    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065726    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065728    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065731    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065734    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065736    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065739    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065741    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065744    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065747    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.076235 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065749    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065752    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065755    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065757    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065760    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065762    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065765    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065768    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065771    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065773    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065776    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065778    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065781    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065783    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065786    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065790    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065792    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065795    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065803    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065807    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.076719 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065810    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065813    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065815    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065818    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065820    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065823    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065826    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065828    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065831    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065834    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065836    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065839    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065842    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065844    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065847    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065849    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065852    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065854    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065857    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.077183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065859    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065862    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065865    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065868    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065870    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065873    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065875    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065878    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065882    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065884    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065887    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065890    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065894    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065897    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065899    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065902    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065904    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065907    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065910    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.077628 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065912    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065915    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065918    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065921    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065924    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065928    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065932    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.065935    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.073174    2567 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.073193    2567 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.073213    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.073221    2567 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073696    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073709    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073713    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073720    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073725    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073729    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073732    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.078147 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073735    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073738    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073740    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073743    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073746    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073749    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073751    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073754    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073757    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073760    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073763    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073766    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073768    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073771    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073774    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073779    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073784    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073787    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073790    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.078618 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073794    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073799    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073804    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073811    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073815    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073817    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073822    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073825    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073828    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073830    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073834    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073836    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073839    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073842    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073845    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073848    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073850    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073855    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073857    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073860    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.079088 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073865    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073869    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073874    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073878    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073883    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073888    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073893    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073897    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073900    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073903    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073906    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073909    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073911    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073914    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073916    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073919    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073923    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073925    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073928    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.079590 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073931    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073934    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073937    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073940    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073943    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073945    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073948    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073951    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073956    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073962    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073966    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073969    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073972    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073974    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073978    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073981    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073984    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073987    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073990    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073992    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.080040 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073995    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.073998    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074000    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074003    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074006    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074008    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074011    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074014    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074016    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074019    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074021    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074024    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074027    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074030    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074035    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074039    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074043    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074046    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074050    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.080507 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074053    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074056    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.074085    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074277    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074283    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074286    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074288    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074291    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074294    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074297    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074299    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074302    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074305    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074307    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074310    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.081003 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074313    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074316    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074318    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074321    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074324    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074326    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074329    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074332    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074336    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074341    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074346    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074350    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074354    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074356    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074359    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074362    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074365    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074367    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074370    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.081355 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074372    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074375    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074377    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074380    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074383    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074385    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074388    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074391    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074394    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074396    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074399    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074401    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074404    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074407    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074409    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074413    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074417    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074420    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074423    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074426    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.081808 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074429    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074431    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074435    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074439    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074443    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074448    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074451    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074454    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074456    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074459    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074461    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074464    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074466    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074469    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074471    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074473    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074476    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074479    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074481    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074484    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.082266 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074487    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074490    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074493    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074495    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074498    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074500    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074503    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074506    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074508    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074511    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074513    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074516    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074518    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074521    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074524    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074526    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074529    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074531    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074534    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074537    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.082751 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074539    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074542    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074544    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074547    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074550    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074553    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074555    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074558    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074560    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074563    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074565    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074574    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074579    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074584    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074587    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.074613    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:47:06.083205 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074663    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074668    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074671    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074675    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074678    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074681    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074684    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074687    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074689    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074692    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074694    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074698    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074700    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074703    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074705    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074708    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074711    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074713    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074716    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:47:06.083572 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074718    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074721    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074724    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074728    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074731    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074734    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074736    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074739    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074741    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074744    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074746    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074750    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074753    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074756    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074758    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074761    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074763    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074766    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074768    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:47:06.084015 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074772    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074776    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074779    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074782    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074785    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074787    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074790    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074795    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074799    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074803    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074807    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074809    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074812    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074814    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074817    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074820    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074824    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074827    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074830    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074832    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:47:06.084589 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074835    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074837    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074840    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074842    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074845    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074848    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074851    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074855    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074858    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074861    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074863    2567 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074866    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074868    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074871    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074874    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074876    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074879    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074881    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074884    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:47:06.085169 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074886    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074889    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074891    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074894    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074897    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074899    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074902    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074905    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074907    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074911    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074914    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074917    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074919    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074923    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074925    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074928    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074930    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074933    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074935    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074939    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:47:06.085625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074941    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074944    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074947    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074949    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074952    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074954    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074957    2567 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074959    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:06.074962    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.075019    2567 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.078703    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.079361    2567 server.go:1014] "Starting client certificate rotation"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.079451    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.079485    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.080836    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:47:06.086202 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.081995    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:47:06.090537 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.090520    2567 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:47:06.093411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.093398    2567 log.go:25] "Validated CRI v1 image API"
Nov 03 20:47:06.094169 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.094142    2567 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:47:06.096116 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.096098    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad0b69a6-16ee-44ee-97ca-ec4ecb5df7a0:/dev/nvme0n1p3 cffd87f9-d003-4e32-9fed-649586e51184:/dev/nvme0n1p4]
Nov 03 20:47:06.096182 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.096115    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:47:06.101584 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.101470    2567 manager.go:217] Machine: {Timestamp:2026-11-03 20:47:06.100332156 +0000 UTC m=+0.244481907 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103438 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2493f0b6a5c3c1512990392abce7ac SystemUUID:ec2493f0-b6a5-c3c1-5129-90392abce7ac BootID:f2de8a4b-0254-47ea-aa44-75260c952258 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d0:9c:16:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d0:9c:16:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:11:8d:44:28:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:47:06.101584 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.101581    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:47:06.101713 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.101655    2567 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:47:06.103910 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.103889    2567 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:47:06.104048 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.103910    2567 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-144.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:47:06.104484 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.104474    2567 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:47:06.104513 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.104485    2567 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:47:06.104513 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.104498    2567 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:47:06.104980 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.104970    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:47:06.105725 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.105715    2567 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:47:06.109505 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.109494    2567 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:47:06.109624 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.109615    2567 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:47:06.110140 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110131    2567 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:47:06.110169 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110143    2567 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:47:06.110169 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110152    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:47:06.110169 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110161    2567 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:47:06.110266 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110177    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:47:06.110551 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.110538    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:47:06.112353 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.112328    2567 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:47:06.118583 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.118555    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:47:06.119199 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119186    2567 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:47:06.119250 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119216    2567 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:47:06.119364 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119352    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:47:06.119403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119370    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:47:06.119403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119377    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:47:06.119403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119382    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:47:06.119403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119388    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:47:06.119403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119394    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:47:06.119802 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119793    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:47:06.119832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119803    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:47:06.119832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119810    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:47:06.119832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119816    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:47:06.119832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119825    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:47:06.119832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.119833    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:47:06.120382 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.120373    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:47:06.120382 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.120382    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:47:06.123528 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.123515    2567 server.go:1263] "Started kubelet"
Nov 03 20:47:06.123659 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.123600    2567 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:47:06.123709 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.123611    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:47:06.123709 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.123698    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:47:06.124141 ip-10-0-140-144 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:47:06.124930 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.124696    2567 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:47:06.128306 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.128289    2567 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:47:06.129428 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.129408    2567 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:47:06.130605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.130587    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:47:06.131371 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131352    2567 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:47:06.131732 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.131688    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:47:06.131824 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131810    2567 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:47:06.131824 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131820    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:47:06.131899 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131828    2567 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:47:06.131951 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131920    2567 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:47:06.131951 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.131927    2567 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:47:06.132338 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132320    2567 factory.go:55] Registering systemd factory
Nov 03 20:47:06.132338 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132338    2567 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:47:06.132539 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132530    2567 factory.go:153] Registering CRI-O factory
Nov 03 20:47:06.132539 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132540    2567 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:47:06.132610 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132594    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:47:06.132672 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132616    2567 factory.go:103] Registering Raw factory
Nov 03 20:47:06.132672 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.132650    2567 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:47:06.133556 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.133539    2567 manager.go:319] Starting recovery of all containers
Nov 03 20:47:06.145334 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.144084    2567 manager.go:324] Recovery completed
Nov 03 20:47:06.149735 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.149722    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.151599 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.151577    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.151679 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.151612    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.151679 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.151623    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.152156 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.152145    2567 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:47:06.152156 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.152155    2567 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:47:06.152240 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.152158    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.152240 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.152170    2567 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:47:06.153019 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.153001    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.153101 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.153034    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.153101 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.153049    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.232379    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.270358    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.271476    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.271495    2567 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.271550    2567 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.271595    2567 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.272225    2567 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-144.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.272272    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-144.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.272026    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-144.us-west-2.compute.internal.18e4a3f56eeab63e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-144.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-144.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:47:06.123490878 +0000 UTC m=+0.267640630,LastTimestamp:2026-11-03 20:47:06.123490878 +0000 UTC m=+0.267640630,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-144.us-west-2.compute.internal,}"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.298165    2567 policy_none.go:50] "Start"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.298180    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:47:06.298917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.298192    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:47:06.300961 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.300367    2567 policy_none.go:44] "Start"
Nov 03 20:47:06.333430 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.333315    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:47:06.337172 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.337147    2567 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:47:06.339443 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339430    2567 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:47:06.339544 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.339463    2567 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:47:06.339544 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339474    2567 server.go:87] "Starting device plugin registration server"
Nov 03 20:47:06.339774 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339761    2567 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:47:06.339832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339775    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:47:06.340164 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339921    2567 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:47:06.340164 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339991    2567 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:47:06.340164 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.339998    2567 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:47:06.340403 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.340385    2567 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:47:06.340460 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.340434    2567 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:47:06.341817 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.341797    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:47:06.341892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.341867    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8g9cl"
Nov 03 20:47:06.372360 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.372339    2567 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"]
Nov 03 20:47:06.372426 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.372412    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.375892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.375878    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.375942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.375906    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.375942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.375922    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.378109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378098    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.378253 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378241    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.378288 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378267    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.378865 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378848    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.378959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378864    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.378959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378878    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.378959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378888    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.378959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378894    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.378959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.378905    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.381001 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.380985    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.381081 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.381014    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.382396 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.382382    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.382496 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.382409    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.382496 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.382425    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.399972 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.399955    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.402262 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.402247    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.420998 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.420978    2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:06.422741 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.422725    2567 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:06.433472 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.433452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.433532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.433490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.433580 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.433536    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.440613 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.440597    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:47:06.441316 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.441297    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:47:06.441393 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.441323    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:47:06.441393 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.441335    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:47:06.441393 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.441357    2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.513561 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.513535    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:06.514132 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.514115    2567 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.514174 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:06.514139    2567 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-144.us-west-2.compute.internal\": node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:47:06.514408 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.514396    2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:06.532212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.532194    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534364 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534437 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534437 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534467    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534470    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.534532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.534467    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.596695 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.596647    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:47:06.597390 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.597377    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.653515 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.653495    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:47:06.702945 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.702927    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:06.705090 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:06.705077    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:47:07.079309 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.079233    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:47:07.079859 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.079396    2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:47:07.079859 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.079356    2567 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7b0dd8fedf904086887a3a80115b944-867db6f534d550a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-144.us-west-2.compute.internal.18e4a3f570979cd6\": read tcp 10.0.140.144:56608->18.220.87.170:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-144.us-west-2.compute.internal.18e4a3f570979cd6  default   5195 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-144.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-144.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-144.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:47:06 +0000 UTC,LastTimestamp:2026-11-03 20:47:06.378862959 +0000 UTC m=+0.523012712,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-144.us-west-2.compute.internal,}"
Nov 03 20:47:07.079859 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.079396    2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:47:07.079859 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.079441    2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:47:07.111213 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.111195    2567 apiserver.go:52] "Watching apiserver"
Nov 03 20:47:07.130941 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.130919    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:47:07.346968 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.346903    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:42:06 +0000 UTC" deadline="2028-04-05 19:38:03.480024307 +0000 UTC"
Nov 03 20:47:07.346968 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.346927    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12454h50m56.133099451s"
Nov 03 20:47:07.372942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.372923    2567 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:47:07.373255 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.373238    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vdkpg","openshift-multus/network-metrics-daemon-fjkp6","openshift-ovn-kubernetes/ovnkube-node-n99n6","kube-system/konnectivity-agent-qj2ct","openshift-cluster-node-tuning-operator/tuned-9thx7","openshift-multus/multus-8kvc5","kube-system/global-pull-secret-syncer-stvfh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn","openshift-network-diagnostics/network-check-target-h4tkw","openshift-dns/node-resolver-cwmvz","openshift-image-registry/node-ca-mwqr2","openshift-network-operator/iptables-alerter-s7hmx","kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"]
Nov 03 20:47:07.375735 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.375717    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.377828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.377811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.377904 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.377867    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:07.380000 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.379982    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.382118 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.382099    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.384398 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.384376    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.384475 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.384446    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:07.386601 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.386581    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.388823 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.388805    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.390959 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.390941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.393104 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.393084    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:07.393186 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.393139    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:07.395243 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.395226    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.397422 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.397402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.399847 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.399831    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.427320 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427297    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:47:07.427320 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427307    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\""
Nov 03 20:47:07.427451 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427406    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:47:07.427451 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427408    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.427451 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427426    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.427451 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.427439    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:47:07.428897 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.428852    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.428897 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.428870    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.429034 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.428908    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:47:07.429034 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.428979    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\""
Nov 03 20:47:07.433076 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.431578    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.433076 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.431949    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.433076 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.432124    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:47:07.433076 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.432166    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\""
Nov 03 20:47:07.433833 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.433739    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:47:07.435421 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.435404    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\""
Nov 03 20:47:07.435524 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.435476    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:47:07.437664 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.437646    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.437773 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.437746    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.439999 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.439984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvpg\" (UniqueName: \"kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.440061 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.440061 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.440061 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.440061 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440057    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.440194 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb79\" (UniqueName: \"kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.440194 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440154    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440194 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440291 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.440291 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.440291 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440421 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440421 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440421 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440637 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs54\" (UniqueName: \"kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.440637 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.440637 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440544    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.440637 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.440637 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440622    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4nr\" (UniqueName: \"kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440678    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.440828 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69x2\" (UniqueName: \"kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82gp\" (UniqueName: \"kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.440953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:07.441031 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441027    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441097    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshvv\" (UniqueName: \"kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441465 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441465 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441317    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441465 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441342    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5g6d\" (UniqueName: \"kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441465 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.441465 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441494    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441561    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.441605 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441603    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441660    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441688    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.441832 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.441998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442053    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442154    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.442367 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442353    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzzc\" (UniqueName: \"kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.442566 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.442566 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.442415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.453018 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.453000    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.453193 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.453172    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:47:07.453193 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.453186    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.471455 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.471432    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\""
Nov 03 20:47:07.471455 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.471429    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\""
Nov 03 20:47:07.477538 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477522    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\""
Nov 03 20:47:07.477661 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477648    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\""
Nov 03 20:47:07.477744 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477684    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:47:07.477744 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477706    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:47:07.477744 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477739    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.477877 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477742    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:47:07.477877 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477771    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\""
Nov 03 20:47:07.477877 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.477775    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:47:07.478872 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.478860    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:47:07.479104 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.479087    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:47:07.479155 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.479116    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.479155 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.479143    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:47:07.485977 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.485960    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:47:07.486063 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.486043    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:47:07.486133 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.486118    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j9q7k"
Nov 03 20:47:07.542775 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.542775 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.542923 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69x2\" (UniqueName: \"kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.542923 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.542923 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82gp\" (UniqueName: \"kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.542923 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.542923 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.542978    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543011    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543037    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543128    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543171 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543170    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543215    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshvv\" (UniqueName: \"kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5g6d\" (UniqueName: \"kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543303    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543311    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543314    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.543301    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543358    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543306    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543374    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.543389    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:08.043373703 +0000 UTC m=+2.187523460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.543650 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543430    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543435    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543489    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543522    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543536    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543678    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543734    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543765    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.544405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543895    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543894    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543900    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543954    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543957    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543944    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543963    2567 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.543992    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544021    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544029    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544128    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.544138    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544168    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544181    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544237    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.545206 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544269    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544294    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.544317    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:08.044299591 +0000 UTC m=+2.188449347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544341    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544341    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544335    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544450    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544481    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544476    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.546219 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544490    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544494    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544524    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544693    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544740    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544808    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544868    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzzc\" (UniqueName: \"kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.546874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544968    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvpg\" (UniqueName: \"kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.544989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545067    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545080    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb79\" (UniqueName: \"kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545104    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545114    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545148    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545217    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545333    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.547404 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545321    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545366    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs54\" (UniqueName: \"kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545434    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545494    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545506    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545069    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4nr\" (UniqueName: \"kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545634    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545692    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545737    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545746    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545754    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545280    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545808    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545830    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545856    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.545983    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546002    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546004    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546027    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546058    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546077    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546365    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.546451    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.548025    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.548183    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.548262    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.548671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.548518    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.602923 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.602871    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:07.602923 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.602891    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:07.602923 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.602900    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:07.603122 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:07.602953    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:08.102941817 +0000 UTC m=+2.247091589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:07.605394 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.605267    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82gp\" (UniqueName: \"kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:07.605498 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.605398    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69x2\" (UniqueName: \"kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.605659 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.605634    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5g6d\" (UniqueName: \"kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.605850 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.605834    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshvv\" (UniqueName: \"kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.606489 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.606460    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzzc\" (UniqueName: \"kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.606972 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.606956    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvpg\" (UniqueName: \"kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.607038 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.607024    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs54\" (UniqueName: \"kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.607567 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.607551    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4nr\" (UniqueName: \"kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.609937 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.609919    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb79\" (UniqueName: \"kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.723626 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.723608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:47:07.723626 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.723622    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:47:07.742803 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.742780    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:47:07.753855 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.753834    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kvc5"
Nov 03 20:47:07.768455 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.768432    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:47:07.774452 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.774436    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:07.790278 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.790257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:47:07.803981 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.803961    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:47:07.823892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.823869    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:07.974941 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:07.974910    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b940c9766e72993f1a0cb04738b1af3.slice/crio-e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378 WatchSource:0}: Error finding container e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378: Status 404 returned error can't find the container with id e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378
Nov 03 20:47:07.979385 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:07.979365    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:47:07.984991 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:07.984970    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d51c0bbdc067fa97bdab078f3d2d6ac.slice/crio-22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3 WatchSource:0}: Error finding container 22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3: Status 404 returned error can't find the container with id 22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3
Nov 03 20:47:08.041500 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.041478    2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:08.050315 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.050292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:08.050390 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.050328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:08.050440 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.050428    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:08.050509 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.050434    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:08.050509 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.050476    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:09.050461022 +0000 UTC m=+3.194610764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:08.050509 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.050495    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:09.050484237 +0000 UTC m=+3.194633979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:08.151393 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.151369    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:08.151862 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.151462    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:08.151862 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.151476    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:08.151862 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.151485    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:08.151862 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:08.151524    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:09.151513522 +0000 UTC m=+3.295663276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:08.188511 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.188488    2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:08.275463 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.275394    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerStarted","Data":"22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3"}
Nov 03 20:47:08.276533 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.276515    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"8b940c9766e72993f1a0cb04738b1af3","Type":"ContainerStarted","Data":"e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378"}
Nov 03 20:47:08.489767 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.489725    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:07 +0000 UTC" deadline="2028-08-05 09:26:37.755435124 +0000 UTC"
Nov 03 20:47:08.489767 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.489761    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15372h39m29.265676243s"
Nov 03 20:47:08.493791 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:08.493778    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:47:08.698787 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.698761    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb57a79e_5114_4c7f_97f5_23b45aebd6e7.slice/crio-b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700 WatchSource:0}: Error finding container b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700: Status 404 returned error can't find the container with id b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700
Nov 03 20:47:08.718364 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.718343    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c429b7_91ff_41b2_96fc_575e8cc12640.slice/crio-3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83 WatchSource:0}: Error finding container 3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83: Status 404 returned error can't find the container with id 3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83
Nov 03 20:47:08.788602 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.788573    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa08991_01cf_45cd_a970_0dba2a38a851.slice/crio-5b2aed49f2ef12a640c7520efaa63e321d9f9be0c6ec72b42e4c1d8264a705c7 WatchSource:0}: Error finding container 5b2aed49f2ef12a640c7520efaa63e321d9f9be0c6ec72b42e4c1d8264a705c7: Status 404 returned error can't find the container with id 5b2aed49f2ef12a640c7520efaa63e321d9f9be0c6ec72b42e4c1d8264a705c7
Nov 03 20:47:08.804491 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.804467    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c9b27_49d4_4d22_9ff7_350c94e96665.slice/crio-5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9 WatchSource:0}: Error finding container 5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9: Status 404 returned error can't find the container with id 5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9
Nov 03 20:47:08.811206 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.811186    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6418ffe1_921f_4045_9e38_761687df3ad7.slice/crio-72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611 WatchSource:0}: Error finding container 72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611: Status 404 returned error can't find the container with id 72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611
Nov 03 20:47:08.838324 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.838294    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983d3627_42d7_4192_ba09_c44043b657ee.slice/crio-c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162 WatchSource:0}: Error finding container c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162: Status 404 returned error can't find the container with id c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162
Nov 03 20:47:08.859070 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.859049    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb59a160_8501_4557_b55e_e4f3e87cfa61.slice/crio-4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24 WatchSource:0}: Error finding container 4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24: Status 404 returned error can't find the container with id 4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24
Nov 03 20:47:08.911350 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.911322    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fc55c3_89a7_4d55_bae4_a64d15efebaa.slice/crio-b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31 WatchSource:0}: Error finding container b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31: Status 404 returned error can't find the container with id b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31
Nov 03 20:47:08.912021 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:08.911999    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fd3340_3104_450d_a434_0e89d73d7331.slice/crio-067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b WatchSource:0}: Error finding container 067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b: Status 404 returned error can't find the container with id 067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b
Nov 03 20:47:09.057354 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.057291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:09.057354 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.057336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:09.057512 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.057450    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:09.057512 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.057498    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:11.057482456 +0000 UTC m=+5.201632222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:09.057512 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.057498    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:09.057716 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.057574    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:11.057547664 +0000 UTC m=+5.201697425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:09.157868 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.157843    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:09.158294 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.157995    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:09.158294 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.158013    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:09.158294 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.158023    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:09.158294 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.158062    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:11.158051518 +0000 UTC m=+5.302201263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:09.272531 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.272504    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:09.272531 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.272525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:09.272724 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.272625    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:09.272724 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.272628    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:09.272796 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.272732    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:09.272840 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:09.272810    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:09.279937 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.279911    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31"}
Nov 03 20:47:09.281547 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.281521    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kvc5" event={"ID":"8b1c9b27-49d4-4d22-9ff7-350c94e96665","Type":"ContainerStarted","Data":"5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9"}
Nov 03 20:47:09.283024 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.282999    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9thx7" event={"ID":"3fa08991-01cf-45cd-a970-0dba2a38a851","Type":"ContainerStarted","Data":"5b2aed49f2ef12a640c7520efaa63e321d9f9be0c6ec72b42e4c1d8264a705c7"}
Nov 03 20:47:09.284328 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.284289    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s7hmx" event={"ID":"bb57a79e-5114-4c7f-97f5-23b45aebd6e7","Type":"ContainerStarted","Data":"b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700"}
Nov 03 20:47:09.285562 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.285538    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qj2ct" event={"ID":"33fd3340-3104-450d-a434-0e89d73d7331","Type":"ContainerStarted","Data":"067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b"}
Nov 03 20:47:09.286862 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.286835    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwqr2" event={"ID":"eb59a160-8501-4557-b55e-e4f3e87cfa61","Type":"ContainerStarted","Data":"4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24"}
Nov 03 20:47:09.287969 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.287937    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwmvz" event={"ID":"983d3627-42d7-4192-ba09-c44043b657ee","Type":"ContainerStarted","Data":"c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162"}
Nov 03 20:47:09.289261 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.289236    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerStarted","Data":"72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611"}
Nov 03 20:47:09.290512 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.290489    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn" event={"ID":"13c429b7-91ff-41b2-96fc-575e8cc12640","Type":"ContainerStarted","Data":"3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83"}
Nov 03 20:47:09.490459 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.489927    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:07 +0000 UTC" deadline="2028-06-25 06:21:50.301700563 +0000 UTC"
Nov 03 20:47:09.490459 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:09.489983    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14385h34m40.811721739s"
Nov 03 20:47:11.070260 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.070218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:11.070929 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.070286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:11.070929 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.070442    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:11.070929 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.070504    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:15.070486192 +0000 UTC m=+9.214635948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:11.070929 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.070574    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:11.070929 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.070612    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:15.070601024 +0000 UTC m=+9.214750780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:11.171585 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.171526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:11.171805 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.171786    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:11.171887 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.171813    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:11.171887 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.171827    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:11.171887 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.171886    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:15.171867609 +0000 UTC m=+9.316017370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:11.272088 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.272048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:11.272294 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.272177    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:11.272294 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.272244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:11.272438 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.272331    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:11.272438 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:11.272375    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:11.272541 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:11.272444    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:12.309997 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:12.308791    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d51c0bbdc067fa97bdab078f3d2d6ac" containerID="2d41d947ce373a785aabb13bf1f0ab9be91c046c968b53aaf70bd01699928bdb" exitCode=0
Nov 03 20:47:12.309997 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:12.309073    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerDied","Data":"2d41d947ce373a785aabb13bf1f0ab9be91c046c968b53aaf70bd01699928bdb"}
Nov 03 20:47:13.272232 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.272155    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:13.272416 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:13.272294    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:13.272416 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.272371    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:13.272542 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:13.272454    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:13.272542 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.272515    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:13.272643 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:13.272580    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:13.316290 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.316255    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerStarted","Data":"b059236f4c6caef6ac18f185dd40b5733291de82603efbe20cceab9903866e99"}
Nov 03 20:47:13.324968 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.324936    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"8b940c9766e72993f1a0cb04738b1af3","Type":"ContainerStarted","Data":"e5d423321872442a89e991e6d835a2b233c8f94a4db514b53a926c5fab3374e5"}
Nov 03 20:47:13.328610 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.328496    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="7c3a6261e7061f0d3d2f1c2a4f099c538385ec976fd1b4a3251907f232423810" exitCode=0
Nov 03 20:47:13.328610 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.328533    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"7c3a6261e7061f0d3d2f1c2a4f099c538385ec976fd1b4a3251907f232423810"}
Nov 03 20:47:13.437077 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.437006    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" podStartSLOduration=6.436986474 podStartE2EDuration="6.436986474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:13.43670023 +0000 UTC m=+7.580850020" watchObservedRunningTime="2026-11-03 20:47:13.436986474 +0000 UTC m=+7.581136237"
Nov 03 20:47:13.683371 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:13.683256    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal" podStartSLOduration=6.683234762 podStartE2EDuration="6.683234762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:13.553288084 +0000 UTC m=+7.697437861" watchObservedRunningTime="2026-11-03 20:47:13.683234762 +0000 UTC m=+7.827384529"
Nov 03 20:47:14.351838 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:14.350380    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qj2ct" event={"ID":"33fd3340-3104-450d-a434-0e89d73d7331","Type":"ContainerStarted","Data":"8ac0607a376dd2d4decf6a02ea492231db9243c2f110dd851a074e0208ce7072"}
Nov 03 20:47:15.105530 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.105442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:15.105530 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.105500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:15.105764 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.105662    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:15.106307 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.105871    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.105852131 +0000 UTC m=+17.250001884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:15.106307 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.105747    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:15.106307 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.106231    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.106214912 +0000 UTC m=+17.250364671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:15.206919 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.206573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:15.206919 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.206787    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:15.206919 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.206808    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:15.206919 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.206822    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:15.206919 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.206882    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:23.206863852 +0000 UTC m=+17.351013612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.271996    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.272127    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.272475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.272564    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.272626    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:15.272768 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:15.272716    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:15.356021 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.355935    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mwqr2" event={"ID":"eb59a160-8501-4557-b55e-e4f3e87cfa61","Type":"ContainerStarted","Data":"863214f60682976dec80b4fb89531f98fb0f3ea7779cc6a25bb53e8c00d250ab"}
Nov 03 20:47:15.464197 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.463303    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qj2ct" podStartSLOduration=3.387673708 podStartE2EDuration="8.463283255s" totalImagesPullingTime="5.075609547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:14.463941252 +0000 UTC m=+8.608091016" watchObservedRunningTime="2026-11-03 20:47:15.463283255 +0000 UTC m=+9.607433018"
Nov 03 20:47:15.464197 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:15.463833    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mwqr2" podStartSLOduration=2.519044715 podStartE2EDuration="8.463823301s" totalImagesPullingTime="5.944778586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:15.463061496 +0000 UTC m=+9.607211260" watchObservedRunningTime="2026-11-03 20:47:15.463823301 +0000 UTC m=+9.607973063"
Nov 03 20:47:16.358713 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:16.358605    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn" event={"ID":"13c429b7-91ff-41b2-96fc-575e8cc12640","Type":"ContainerStarted","Data":"f7dddbae135fc0b866e3ed2f6e908922c0ed9baf81a760db4a8e30b00ccfc6eb"}
Nov 03 20:47:17.272050 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:17.272016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:17.272223 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:17.272139    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:17.272568 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:17.272548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:17.272719 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:17.272658    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:17.272799 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:17.272773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:17.272875 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:17.272860    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:17.740921 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:17.740856    2567 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:47:18.351028 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.350766    2567 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:47:17.740874763Z","UUID":"97bcd6c3-a96d-433a-89e8-06ba0635c2bd","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:47:18.352529 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.352507    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:47:18.352529 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.352536    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:47:18.364064 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.364031    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwmvz" event={"ID":"983d3627-42d7-4192-ba09-c44043b657ee","Type":"ContainerStarted","Data":"b6181e2ee18c77357cd0338f2df55d10f53128e24bfdac72dd1f722c461eac3b"}
Nov 03 20:47:18.366115 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.366090    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="42598fd81d52ca02fed314fe3570497776f446601572137ad51d045cb293f0b8" exitCode=0
Nov 03 20:47:18.366227 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.366168    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"42598fd81d52ca02fed314fe3570497776f446601572137ad51d045cb293f0b8"}
Nov 03 20:47:18.368324 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.368302    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn" event={"ID":"13c429b7-91ff-41b2-96fc-575e8cc12640","Type":"ContainerStarted","Data":"ca52ea89ff751e10b87a1ac5d539fc24174df5325453ba35c6dc62e840a2a5d9"}
Nov 03 20:47:18.370262 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.370238    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kvc5" event={"ID":"8b1c9b27-49d4-4d22-9ff7-350c94e96665","Type":"ContainerStarted","Data":"00971a97a35b295279a2a15af42f51c7df827a02d8398c1e016767d1c9a71892"}
Nov 03 20:47:18.511826 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.511766    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cwmvz" podStartSLOduration=2.763076003 podStartE2EDuration="11.511747099s" totalImagesPullingTime="8.748671096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:18.510714326 +0000 UTC m=+12.654864087" watchObservedRunningTime="2026-11-03 20:47:18.511747099 +0000 UTC m=+12.655896863"
Nov 03 20:47:18.622684 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.622610    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8kvc5" podStartSLOduration=3.15025733 podStartE2EDuration="11.622594117s" totalImagesPullingTime="8.472336787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:18.622366935 +0000 UTC m=+12.766516697" watchObservedRunningTime="2026-11-03 20:47:18.622594117 +0000 UTC m=+12.766743879"
Nov 03 20:47:18.761221 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.761153    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:18.761738 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:18.761717    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:19.272152 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.272121    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:19.272312 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:19.272226    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:19.272579 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.272561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:19.272699 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:19.272665    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:19.272812 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.272729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:19.272812 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:19.272801    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:19.375337 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.375307    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s7hmx" event={"ID":"bb57a79e-5114-4c7f-97f5-23b45aebd6e7","Type":"ContainerStarted","Data":"b65cb9b50ffcf4599b3192d44a6c0ce9c06e2832a89a2ca85c0496ffa98aa1d4"}
Nov 03 20:47:19.375444 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.375368    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:19.376436 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.376418    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:47:19.483946 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:19.483901    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s7hmx" podStartSLOduration=4.332600992 podStartE2EDuration="13.483886863s" totalImagesPullingTime="9.151285871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:19.483350634 +0000 UTC m=+13.627500398" watchObservedRunningTime="2026-11-03 20:47:19.483886863 +0000 UTC m=+13.628036624"
Nov 03 20:47:20.378823 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.378793    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="f4a1fab13231c7735063ff5020b6db68cef2776228d568598898493484d249b0" exitCode=0
Nov 03 20:47:20.379607 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.378874    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"f4a1fab13231c7735063ff5020b6db68cef2776228d568598898493484d249b0"}
Nov 03 20:47:20.381109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.381085    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn" event={"ID":"13c429b7-91ff-41b2-96fc-575e8cc12640","Type":"ContainerStarted","Data":"e64907d00238c1d2dd843ce4c1c7da0b6fa17b6ac54463269c31b6fa911cddb6"}
Nov 03 20:47:20.382840 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.382795    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9thx7" event={"ID":"3fa08991-01cf-45cd-a970-0dba2a38a851","Type":"ContainerStarted","Data":"933a35519ed6ad2ca7207d118bb05d423d5488b56af61a60cad709dd6caed072"}
Nov 03 20:47:20.604484 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.604440    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn" podStartSLOduration=4.378076958 podStartE2EDuration="14.604426744s" totalImagesPullingTime="10.226349786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:20.603767413 +0000 UTC m=+14.747917175" watchObservedRunningTime="2026-11-03 20:47:20.604426744 +0000 UTC m=+14.748576504"
Nov 03 20:47:20.715086 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:20.714989    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9thx7" podStartSLOduration=2.866857998 podStartE2EDuration="13.714975209s" totalImagesPullingTime="10.848117211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:20.714416635 +0000 UTC m=+14.858566395" watchObservedRunningTime="2026-11-03 20:47:20.714975209 +0000 UTC m=+14.859124970"
Nov 03 20:47:21.271780 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:21.271752    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:21.271935 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:21.271845    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:21.271935 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:21.271858    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:21.271935 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:21.271928    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:21.272075 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:21.272022    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:21.272166 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:21.272147    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:22.390010 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:22.389809    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="bcce2f274f153f4d3cea3f363c9e76e5ca1484fa6e0f7ff4eacd52da6bd96c4d" exitCode=0
Nov 03 20:47:22.390010 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:22.389889    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"bcce2f274f153f4d3cea3f363c9e76e5ca1484fa6e0f7ff4eacd52da6bd96c4d"}
Nov 03 20:47:23.171782 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.171745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:23.172004 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.171801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:23.172004 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.171908    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:23.172004 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.171912    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:23.172004 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.171964    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.171949256 +0000 UTC m=+33.316099001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:23.172004 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.171986    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.171975429 +0000 UTC m=+33.316125170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:23.272288 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.272262    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:23.272288 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.272274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:23.272288 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.272292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272367    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272385    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272380    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272396    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272434    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:47:39.27242254 +0000 UTC m=+33.416572282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:23.272270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:23.272519 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272455    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:23.272765 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:23.272538    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:24.395651 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:24.395601    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"321f4fb7f31cc513196a80e396dd21163d5fe0ac25e985ac8a6a2a4d66cbe56f"}
Nov 03 20:47:24.395932 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:24.395710    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"7abe639a0d7c230dd20fa2a333b98bef39accbf6133b17201911f8aa25bf9fee"}
Nov 03 20:47:25.272739 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.272501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:25.272870 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.272583    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:25.272870 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:25.272823    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:25.272948 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.272594    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:25.272948 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:25.272914    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:25.273009 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:25.272985    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:25.400400 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400376    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400719    2567 generic.go:356] "Generic (PLEG): container finished" podID="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" containerID="321f4fb7f31cc513196a80e396dd21163d5fe0ac25e985ac8a6a2a4d66cbe56f" exitCode=1
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400796    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerDied","Data":"321f4fb7f31cc513196a80e396dd21163d5fe0ac25e985ac8a6a2a4d66cbe56f"}
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400886    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"4176076514c113bea22978e7a751e9f055bde4e983cf92fa816cf6aaa8d9977c"}
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400940    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"8cce533f86b8b9b45b0e4c294bfac5ee220abd1a8fb9460611ed5ff9993cab55"}
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.400987    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"4ff81537b827032beab44d0909b8ee97fff2e7cd7d9efd03e66acb3c732a1ab4"}
Nov 03 20:47:25.401026 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.401029    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"511180811d1a52508524ae8da55344d4835b043d36f7240644fcfeb22752b344"}
Nov 03 20:47:25.403498 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.403473    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="f99679f76294099b7321a639d20edc8beb9d85529fba9753bae782edbcc405bf" exitCode=0
Nov 03 20:47:25.403609 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:25.403506    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"f99679f76294099b7321a639d20edc8beb9d85529fba9753bae782edbcc405bf"}
Nov 03 20:47:26.408071 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:26.408039    2567 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="a1050e300d85ba05e31e6b42051a8b2f1aff49b258a6ed5fd96962bfd87882cf" exitCode=0
Nov 03 20:47:26.408514 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:26.408089    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerDied","Data":"a1050e300d85ba05e31e6b42051a8b2f1aff49b258a6ed5fd96962bfd87882cf"}
Nov 03 20:47:27.272290 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.272257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:27.272439 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.272348    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:27.272439 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.272429    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:27.272503 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:27.272439    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:27.272533 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:27.272499    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:27.272582 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:27.272564    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:27.413147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.413115    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" event={"ID":"6418ffe1-921f-4045-9e38-761687df3ad7","Type":"ContainerStarted","Data":"b1d9440ed98ddf50bc44e5b8ce5955e0a099754583c1b2d0acf4a2b4d4cfbd13"}
Nov 03 20:47:27.416063 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.416042    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:47:27.416368 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.416351    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"721016fb193425b57860c0cdd97070b099ecc8a2bd60d80a8165ffe21b9b00af"}
Nov 03 20:47:27.528324 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:27.528255    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vdkpg" podStartSLOduration=8.468069533 podStartE2EDuration="21.528243556s" totalImagesPullingTime="13.060174023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:27.527963698 +0000 UTC m=+21.672113459" watchObservedRunningTime="2026-11-03 20:47:27.528243556 +0000 UTC m=+21.672393317"
Nov 03 20:47:29.272491 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.272314    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:29.273050 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.272384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:29.273050 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:29.272559    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:29.273050 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.272430    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:29.273050 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:29.272712    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:29.273050 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:29.272814    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:29.425890 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.425873    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:47:29.426218 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.426201    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"ea96fb8999154f47316f442ab07466bb3639df2d736135d0db0990678310b12b"}
Nov 03 20:47:29.426498 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.426480    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:29.426584 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.426526    2567 scope.go:122] "RemoveContainer" containerID="321f4fb7f31cc513196a80e396dd21163d5fe0ac25e985ac8a6a2a4d66cbe56f"
Nov 03 20:47:29.426584 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.426553    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:29.426704 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.426602    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:29.442365 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.442348    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:29.442464 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:29.442443    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:47:30.432016 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:30.431990    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:47:30.432462 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:30.432388    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" event={"ID":"c8fc55c3-89a7-4d55-bae4-a64d15efebaa","Type":"ContainerStarted","Data":"12d215704b144a23767896880485798ef926f89b66cd602d6c78075328d2d615"}
Nov 03 20:47:30.626424 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:30.626382    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6" podStartSLOduration=8.384148031 podStartE2EDuration="23.626369374s" totalImagesPullingTime="15.242221343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:30.626056278 +0000 UTC m=+24.770206040" watchObservedRunningTime="2026-11-03 20:47:30.626369374 +0000 UTC m=+24.770519135"
Nov 03 20:47:31.271787 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.271753    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:31.271940 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.271859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:31.271940 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.271866    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:31.272049 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.271982    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:31.272095 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.272045    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:31.272142 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.272120    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:31.524722 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.524692    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stvfh"]
Nov 03 20:47:31.525192 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.524997    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:31.525192 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.525107    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:31.527819 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.527788    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4tkw"]
Nov 03 20:47:31.527956 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.527894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:31.528012 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.527990    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:31.528434 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.528412    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjkp6"]
Nov 03 20:47:31.528547 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:31.528500    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:31.528608 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:31.528576    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:32.134931 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:32.134907    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwmvz_983d3627-42d7-4192-ba09-c44043b657ee/dns-node-resolver/0.log"
Nov 03 20:47:33.169551 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:33.169488    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mwqr2_eb59a160-8501-4557-b55e-e4f3e87cfa61/node-ca/0.log"
Nov 03 20:47:33.271802 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:33.271778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:33.271921 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:33.271870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:33.271921 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:33.271877    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:33.271999 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:33.271962    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:33.271999 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:33.271996    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:33.272069 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:33.272042    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:35.272060 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:35.271991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:35.272060 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:35.272019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:35.272489 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:35.272079    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:35.272489 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:35.272088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:35.272489 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:35.272172    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:35.272489 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:35.272276    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:37.272216 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:37.272037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:37.272582 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:37.272091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:37.272582 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:37.272299    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:37.272582 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:37.272098    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:37.272582 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:37.272357    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:37.272582 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:37.272414    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:39.192399 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.192362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:39.192399 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.192405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:39.192933 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.192506    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:39.192933 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.192509    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:39.192933 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.192565    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret podName:c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec nodeName:}" failed. No retries permitted until 2026-11-03 20:48:11.192548076 +0000 UTC m=+65.336697833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret") pod "global-pull-secret-syncer-stvfh" (UID: "c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:47:39.192933 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.192578    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs podName:40fb7803-b57b-4822-b933-41bfa1c40f24 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:11.192572564 +0000 UTC m=+65.336722302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs") pod "network-metrics-daemon-fjkp6" (UID: "40fb7803-b57b-4822-b933-41bfa1c40f24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:47:39.272470 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.272430    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:39.272613 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.272454    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:39.272613 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.272542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:39.272613 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.272549    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:39.272769 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.272609    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:39.272769 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.272696    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:39.293691 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:39.293646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:39.293810 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.293789    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:47:39.293851 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.293809    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:47:39.293851 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.293822    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h2p9g for pod openshift-network-diagnostics/network-check-target-h4tkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:39.293910 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:39.293867    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g podName:c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39 nodeName:}" failed. No retries permitted until 2026-11-03 20:48:11.293854401 +0000 UTC m=+65.438004158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2p9g" (UniqueName: "kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g") pod "network-check-target-h4tkw" (UID: "c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:47:41.272006 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:41.271975    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:41.272415 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:41.272078    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:41.272415 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:41.272093    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:41.272415 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:41.271978    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:41.272415 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:41.272187    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:41.272415 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:41.272249    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:43.272212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:43.272177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:43.272212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:43.272205    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:43.272628 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:43.272274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:43.272628 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:43.272288    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:43.272628 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:43.272340    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:43.272628 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:43.272423    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:45.272860 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:45.272822    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:45.272860 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:45.272855    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:45.273291 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:45.272864    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:45.273291 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:45.272937    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:45.273291 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:45.272979    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:45.273291 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:45.273082    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:47.272478 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:47.272444    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:47.272944 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:47.272546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:47.272944 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:47.272559    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:47.272944 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:47.272620    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:47.272944 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:47.272629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:47.272944 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:47.272732    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:49.272651 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:49.272590    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:49.273107 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:49.272600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:49.273107 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:49.272725    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:49.273107 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:49.272608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:49.273107 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:49.272820    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:49.273107 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:49.272874    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:51.272407 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:51.272373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:51.272823 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:51.272373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:51.272823 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:51.272473    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjkp6" podUID="40fb7803-b57b-4822-b933-41bfa1c40f24"
Nov 03 20:47:51.272823 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:51.272495    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:51.272823 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:51.272551    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4tkw" podUID="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39"
Nov 03 20:47:51.272823 ip-10-0-140-144 kubenswrapper[2567]: E1103 20:47:51.272617    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stvfh" podUID="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec"
Nov 03 20:47:52.181235 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.181211    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:47:52.181384 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.181319    2567 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:47:52.291161 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.291133    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hjll2"]
Nov 03 20:47:52.292861 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.292846    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9sd"]
Nov 03 20:47:52.292973 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.292955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.294810 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.294794    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.335213 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.335189    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hjll2"]
Nov 03 20:47:52.337918 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.337902    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9sd"]
Nov 03 20:47:52.342086 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.342067    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:47:52.342304 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.342284    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\""
Nov 03 20:47:52.342440 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.342292    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:47:52.377354 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.377334    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bznhk"]
Nov 03 20:47:52.379092 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.379079    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.384825 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.384801    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.384915 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.384835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.384915 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.384851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sgjk\" (UniqueName: \"kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.384993 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.384948    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.385035 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.385016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.385076 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.385063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.385129 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.385115    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.385190 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.385175    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.385229 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.385199    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r6x\" (UniqueName: \"kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.437190 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.437133    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bznhk"]
Nov 03 20:47:52.442711 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442685    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:47:52.442711 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442692    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:47:52.442711 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442692    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:47:52.442711 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442706    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:47:52.442942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442706    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:47:52.442942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442743    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\""
Nov 03 20:47:52.442942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442834    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:47:52.442942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442862    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:47:52.442942 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.442900    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\""
Nov 03 20:47:52.486250 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486336 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.486336 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r6x\" (UniqueName: \"kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.486336 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486336 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486321    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486342    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgjk\" (UniqueName: \"kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz4b\" (UniqueName: \"kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486466    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.486889 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486743    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.486889 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486880    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.487025 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486998    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.487069 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.486996    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.490029 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.490012    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.490109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.490063    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.547520 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.547491    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgjk\" (UniqueName: \"kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.547598 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.547542    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r6x\" (UniqueName: \"kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.587701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.587682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz4b\" (UniqueName: \"kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.587799 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.587719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.589592 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.589571    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.605364 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.605345    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:52.610675 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.610661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:47:52.645652 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.645611    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz4b\" (UniqueName: \"kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:52.688533 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:52.688483    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:47:53.029494 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.029432    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hjll2"]
Nov 03 20:47:53.032454 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.032431    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9sd"]
Nov 03 20:47:53.033210 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:53.033188    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c813be_5ca7_4418_891f_04939346e716.slice/crio-64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17 WatchSource:0}: Error finding container 64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17: Status 404 returned error can't find the container with id 64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17
Nov 03 20:47:53.070231 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:53.070210    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d8677f_d924_45ce_8ff2_75fe9725346c.slice/crio-2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33 WatchSource:0}: Error finding container 2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33: Status 404 returned error can't find the container with id 2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33
Nov 03 20:47:53.078268 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.078250    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bznhk"]
Nov 03 20:47:53.080929 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:53.080908    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eda94b8_a813_448e_8d29_742cd73e5d80.slice/crio-fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a WatchSource:0}: Error finding container fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a: Status 404 returned error can't find the container with id fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a
Nov 03 20:47:53.272215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.272190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:47:53.272309 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.272265    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:47:53.272361 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.272348    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.321756    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.322311    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.322437    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.322505    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.322779    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.322973    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\""
Nov 03 20:47:53.324532 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.323008    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:47:53.477664 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.477619    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bznhk" event={"ID":"7eda94b8-a813-448e-8d29-742cd73e5d80","Type":"ContainerStarted","Data":"fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a"}
Nov 03 20:47:53.478756 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.478735    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hjll2" event={"ID":"56c813be-5ca7-4418-891f-04939346e716","Type":"ContainerStarted","Data":"64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17"}
Nov 03 20:47:53.480153 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.480131    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9sd" event={"ID":"c7d8677f-d924-45ce-8ff2-75fe9725346c","Type":"ContainerStarted","Data":"edc9c32c51c4b1df12455c0b53915cebf883dc42bb265334f1c505c8da627aaa"}
Nov 03 20:47:53.480234 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:53.480168    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9sd" event={"ID":"c7d8677f-d924-45ce-8ff2-75fe9725346c","Type":"ContainerStarted","Data":"2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33"}
Nov 03 20:47:55.127581 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.127546    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:47:55.130701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.129607    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.225726 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225693    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-znh84\""
Nov 03 20:47:55.225831 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225740    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:47:55.225831 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225783    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:47:55.225831 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225814    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:47:55.225831 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225814    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:47:55.225831 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225830    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:47:55.226044 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225877    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:47:55.226044 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.225906    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:47:55.227110 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.227093    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:47:55.306915 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.306885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307073 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.306921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307073 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.306967    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307073 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.306989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307073 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.307020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307073 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.307074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.307317 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.307093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408403 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408489 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408489 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408449    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408682 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408755 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408755 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.408850 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.408774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.409217 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.409187    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.409365 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.409311    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.409430 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.409381    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.409874 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.409851    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.411765 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.411743    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.412245 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.412221    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.466901 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.466874    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.487566 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.487500    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hjll2" event={"ID":"56c813be-5ca7-4418-891f-04939346e716","Type":"ContainerStarted","Data":"7bde859484bc05e8f9e981eeb6843d2609f76f6dd31bac4cc13480703fe100a0"}
Nov 03 20:47:55.489916 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.489894    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9sd" event={"ID":"c7d8677f-d924-45ce-8ff2-75fe9725346c","Type":"ContainerStarted","Data":"a9f09d6063f587627cdec4a6cb95fa4cc2abe93a89e78e1cc687664dced1f666"}
Nov 03 20:47:55.655229 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.655139    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"]
Nov 03 20:47:55.657694 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.657668    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.707578 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.707557    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:47:55.707716 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.707663    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ztnjk\""
Nov 03 20:47:55.707822 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.707773    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:47:55.707917 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.707852    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:47:55.708045 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.707942    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:47:55.708045 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.708030    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:47:55.713750 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.713727    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"]
Nov 03 20:47:55.743458 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.743431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:47:55.811988 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.811956    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbvh\" (UniqueName: \"kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.812117 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.812026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.812117 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.812056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.812232 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.812154    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.913391 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.913311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbvh\" (UniqueName: \"kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.913391 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.913377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.913585 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.913400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.913585 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.913455    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.914177 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.914131    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.916908 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.916862    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.916908 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.916885    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:55.972197 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:55.972172    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbvh\" (UniqueName: \"kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:56.112779 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.112743    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:47:56.145183 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:56.145143    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b31527_43de_413d_8b90_962283a47555.slice/crio-2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a WatchSource:0}: Error finding container 2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a: Status 404 returned error can't find the container with id 2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a
Nov 03 20:47:56.271953 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.271926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:47:56.494492 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.494451    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f484b7d7-ptsf7" event={"ID":"f3b31527-43de-413d-8b90-962283a47555","Type":"ContainerStarted","Data":"2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a"}
Nov 03 20:47:56.496596 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.496571    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hjll2" event={"ID":"56c813be-5ca7-4418-891f-04939346e716","Type":"ContainerStarted","Data":"db57b414a51852ef5fd5777f2b52b563f9176ed958fb7ce65b0810fe077e0fd1"}
Nov 03 20:47:56.496783 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.496763    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hjll2"
Nov 03 20:47:56.642769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:56.642738    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"]
Nov 03 20:47:56.649273 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:47:56.649247    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37651c6c_a9ce_4c30_af0f_7e74869a7ef4.slice/crio-86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9 WatchSource:0}: Error finding container 86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9: Status 404 returned error can't find the container with id 86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9
Nov 03 20:47:57.172749 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.172703    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hjll2" podStartSLOduration=2.817659969 podStartE2EDuration="5.172690824s" totalImagesPullingTime="2.355030855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.171816288 +0000 UTC m=+51.315966098" watchObservedRunningTime="2026-11-03 20:47:57.172690824 +0000 UTC m=+51.316840585"
Nov 03 20:47:57.501709 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.501612    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9sd" event={"ID":"c7d8677f-d924-45ce-8ff2-75fe9725346c","Type":"ContainerStarted","Data":"968312b717b7c3f0a2dc8bfd6544f47b90d5ffd77825b6e3d76284feec868fa4"}
Nov 03 20:47:57.503075 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.503020    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9" event={"ID":"37651c6c-a9ce-4c30-af0f-7e74869a7ef4","Type":"ContainerStarted","Data":"86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9"}
Nov 03 20:47:57.504567 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.504531    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bznhk" event={"ID":"7eda94b8-a813-448e-8d29-742cd73e5d80","Type":"ContainerStarted","Data":"efe27d6daa5be1d1a40cc778aa40dda949dc77bbb28d1baf76ebc5757479a616"}
Nov 03 20:47:57.613601 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.613547    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mv9sd" podStartSLOduration=2.139578149 podStartE2EDuration="5.613531389s" totalImagesPullingTime="3.47395324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:47:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.612883493 +0000 UTC m=+51.757033270" watchObservedRunningTime="2026-11-03 20:47:57.613531389 +0000 UTC m=+51.757681154"
Nov 03 20:47:57.721887 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:57.721832    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bznhk" podStartSLOduration=2.283894242 podStartE2EDuration="5.721815739s" totalImagesPullingTime="3.437921497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.721402262 +0000 UTC m=+51.865552049" watchObservedRunningTime="2026-11-03 20:47:57.721815739 +0000 UTC m=+51.865965501"
Nov 03 20:47:59.512221 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:59.512188    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9" event={"ID":"37651c6c-a9ce-4c30-af0f-7e74869a7ef4","Type":"ContainerStarted","Data":"f61fd8fc355df43f8ed5fd8d99b2fa8918cb682dc05f11f746173e73ac414c54"}
Nov 03 20:47:59.512568 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:59.512243    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9" event={"ID":"37651c6c-a9ce-4c30-af0f-7e74869a7ef4","Type":"ContainerStarted","Data":"009d1c7afa5d6268705a2a8acb850c381b3dd87e2890ad1c4c87bbdfa00a3783"}
Nov 03 20:47:59.664314 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:47:59.664249    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9" podStartSLOduration=2.760089587 podStartE2EDuration="4.664232719s" totalImagesPullingTime="1.904143132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:59.663594187 +0000 UTC m=+53.807743950" watchObservedRunningTime="2026-11-03 20:47:59.664232719 +0000 UTC m=+53.808382476"
Nov 03 20:48:00.516112 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:00.516075    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f484b7d7-ptsf7" event={"ID":"f3b31527-43de-413d-8b90-962283a47555","Type":"ContainerStarted","Data":"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"}
Nov 03 20:48:00.628420 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:00.628368    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86f484b7d7-ptsf7" podStartSLOduration=2.272024278 podStartE2EDuration="5.628354729s" totalImagesPullingTime="3.356330451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:00.627070365 +0000 UTC m=+54.771220144" watchObservedRunningTime="2026-11-03 20:48:00.628354729 +0000 UTC m=+54.772504490"
Nov 03 20:48:01.445756 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:01.445727    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:03.200490 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.200455    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ssfh"]
Nov 03 20:48:03.203716 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.203699    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.299461 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.299421    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:48:03.299629 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.299574    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6h6b\""
Nov 03 20:48:03.299629 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.299595    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:48:03.339456 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.339429    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:48:03.362723 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kf96\" (UniqueName: \"kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362723 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362949 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362949 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362949 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362909    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362949 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362928    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.362949 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.362952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.363116 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.363081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.363147 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.363116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kf96\" (UniqueName: \"kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464158    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464178    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464172    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464274    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464411 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464703 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464683    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464779 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464760    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.464866 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.464848    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.466474 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.466457    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.466570 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.466554    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.521304 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.521273    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kf96\" (UniqueName: \"kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.819103 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:03.819043    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:03.831940 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:03.831920    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dfb03f_f653_43e7_9a8a_4a61bbb2220a.slice/crio-8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d WatchSource:0}: Error finding container 8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d: Status 404 returned error can't find the container with id 8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d
Nov 03 20:48:04.532080 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:04.532043    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ssfh" event={"ID":"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a","Type":"ContainerStarted","Data":"8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d"}
Nov 03 20:48:05.519150 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.519068    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:48:05.522369 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.522353    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.540888 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.540855    2567 generic.go:356] "Generic (PLEG): container finished" podID="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" containerID="ddd6161d5314e3c07ffba4227faf477bcd2e7d1b5cd1e21799cf0a8941902463" exitCode=0
Nov 03 20:48:05.541234 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.540917    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ssfh" event={"ID":"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a","Type":"ContainerDied","Data":"ddd6161d5314e3c07ffba4227faf477bcd2e7d1b5cd1e21799cf0a8941902463"}
Nov 03 20:48:05.621671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621622    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:48:05.621671 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621666    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:48:05.621858 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621624    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:48:05.621858 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621828    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:48:05.621858 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621848    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:48:05.621997 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.621981    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:48:05.623205 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.623183    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:48:05.668554 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.668512    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swkc2\""
Nov 03 20:48:05.668792 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.668767    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:48:05.668896 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.668845    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:48:05.669313 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.669298    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:48:05.682434 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.682434 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.682656 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.682762 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.682928 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683019 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.682946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683078 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683022    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683305 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683305 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683240    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683462 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683502 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683541 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.683585 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.683548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.743839 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.743807    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:05.744006 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.743862    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:05.748812 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.748789    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:05.784660 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.784660 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.784660 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.784921 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.784962 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784950    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785016 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.784974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785064 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785064 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785163 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785163 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785118    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785163 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785142    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785305 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785305 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785192    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.785405 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785389    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.786011 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.785691    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.786854 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.786825    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.787657 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.787622    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.787863 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.787836    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.788106 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.788082    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.788170 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.788139    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.788455 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.788433    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.788668 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.788618    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.789046 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.789026    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.789339 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.789320    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.789946 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.789930    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.845376 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.845343    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:05.850110 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:05.850089    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:06.281175 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.281150    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:48:06.284398 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:06.284374    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1195581f_22f9_447f_8f94_c3a08eaa48a1.slice/crio-c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9 WatchSource:0}: Error finding container c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9: Status 404 returned error can't find the container with id c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9
Nov 03 20:48:06.506849 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.506828    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:06.545649 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.545586    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ssfh" event={"ID":"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a","Type":"ContainerStarted","Data":"43e6f633481fb4b22f7911c7e2f29e88c2850c9b1247ec52f5deb8269f650341"}
Nov 03 20:48:06.545977 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.545627    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ssfh" event={"ID":"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a","Type":"ContainerStarted","Data":"9a704ab1d06965a09437b37bb4683285bb6f0e5cbe5bc86f05f6613fbbbe9f44"}
Nov 03 20:48:06.546892 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.546870    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9"}
Nov 03 20:48:06.550592 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.550572    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:06.847688 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:06.847550    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ssfh" podStartSLOduration=2.406198582 podStartE2EDuration="3.847536254s" totalImagesPullingTime="1.377729412s" totalInitContainerRuntime="63.60826ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:06.847160964 +0000 UTC m=+60.991310727" watchObservedRunningTime="2026-11-03 20:48:06.847536254 +0000 UTC m=+60.991686014"
Nov 03 20:48:08.555023 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:08.554983    2567 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="85420996b2d1fef7e51fd505237eda959587a19601509d4abae8f6f595307878" exitCode=0
Nov 03 20:48:08.555455 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:08.555035    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"85420996b2d1fef7e51fd505237eda959587a19601509d4abae8f6f595307878"}
Nov 03 20:48:10.562786 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:10.562760    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"1e6aa412339e22b13b9a9c172abf8c7f731d8a0b51fde823bb4de9bfb1bd44d2"}
Nov 03 20:48:10.563107 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:10.562820    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"16235897dbfb73b5f4fd59b1d80b3719a0e0703f55958fe480d116a6415b7eec"}
Nov 03 20:48:11.227050 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.227003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:11.227221 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.227085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:11.282619 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.282586    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:48:11.282880 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.282863    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:48:11.290067 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.290042    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:11.290172 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.290084    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:11.327858 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.327835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:11.342216 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.342192    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\""
Nov 03 20:48:11.344286 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.344270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:11.376381 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.376363    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:48:11.428398 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.428378    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:48:11.432706 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.432687    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:11.569061 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.568997    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"b783b37f381dfc475c8846752cc3994877808b8cd8fcc4fb9e697be63d84427b"}
Nov 03 20:48:11.569350 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.569061    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"d615301ad3171733421c15b3123d4aa3f2920ba3b4f70529159be406a9baaf66"}
Nov 03 20:48:11.569350 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.569086    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"b1608b39de685bffc58433b152c25b12d3130da9a8fb78785f2ce7e71b2b9e68"}
Nov 03 20:48:11.635525 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.635488    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\""
Nov 03 20:48:11.636933 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.636916    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\""
Nov 03 20:48:11.640173 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.640151    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:11.643330 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.643313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:11.711158 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:11.711132    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjkp6"]
Nov 03 20:48:12.012599 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.009162    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stvfh"]
Nov 03 20:48:12.013625 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:12.013599    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63d20b7_d348_4c0b_9a07_2e2b45ccd0ec.slice/crio-686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f WatchSource:0}: Error finding container 686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f: Status 404 returned error can't find the container with id 686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f
Nov 03 20:48:12.031566 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.031548    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4tkw"]
Nov 03 20:48:12.033948 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:12.033928    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ca13b7_07a9_4ba0_a1c0_50f0d3a18b39.slice/crio-fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3 WatchSource:0}: Error finding container fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3: Status 404 returned error can't find the container with id fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3
Nov 03 20:48:12.343781 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.343702    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:48:12.348199 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.348171    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.399380 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399354    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:48:12.399380 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399369    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:48:12.399380 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399370    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:48:12.399547 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399392    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:48:12.399547 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399403    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q7k9s5blul5n\""
Nov 03 20:48:12.399547 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399536    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:48:12.399658 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399565    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:48:12.399658 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.399584    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:48:12.436215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436319 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436319 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436319 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436319 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436443 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436443 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436443 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436529 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436529 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436529 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436613 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436613 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436707 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436707 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436651    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436707 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436791 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436791 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436853 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436794    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.436853 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.436809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.493167 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.493148    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:48:12.493167 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.493165    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6vrj\""
Nov 03 20:48:12.493329 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.493233    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:48:12.493329 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.493249    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:48:12.493329 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.493184    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:48:12.494997 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.494980    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:48:12.539212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539212 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539397 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.539701 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.539782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540130    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540170    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540178    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540213    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540278    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540345    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:48:12.540669 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540372    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.541138 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.540735    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.542935 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.542913    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.545740 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.543300    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.545740 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.543523    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.545740 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.543963    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.545740 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.545462    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.545740 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.545624    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.546012 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.545875    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.546012 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.545932    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.546445 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.546424    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.546873 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.546852    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.546873 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.546866    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.547100 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.547077    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.547100 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.547095    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.551464 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.551439    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.552965 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.552942    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.573899 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.573872    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stvfh" event={"ID":"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec","Type":"ContainerStarted","Data":"686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f"}
Nov 03 20:48:12.575082 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.575062    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4tkw" event={"ID":"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39","Type":"ContainerStarted","Data":"fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3"}
Nov 03 20:48:12.576165 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.576148    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjkp6" event={"ID":"40fb7803-b57b-4822-b933-41bfa1c40f24","Type":"ContainerStarted","Data":"dbc7e3f16a09cc9e24d6799d6497e61a5c52f19465143e2298e487bac545665e"}
Nov 03 20:48:12.579330 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.579313    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerStarted","Data":"7ffd7bce383178d7c6e69a0ae84c9c8379be065e56345a4158ecef016098883b"}
Nov 03 20:48:12.598291 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.598243    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.677317 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.677296    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:12.704494 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:12.704443    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.01617665 podStartE2EDuration="7.704425574s" totalImagesPullingTime="4.677426466s" totalInitContainerRuntime="10.822458ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:12.702501537 +0000 UTC m=+66.846651299" watchObservedRunningTime="2026-11-03 20:48:12.704425574 +0000 UTC m=+66.848575335"
Nov 03 20:48:13.111489 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.111453    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:48:13.138583 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:13.138541    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943e14bc_5d36_4152_8049_585724259f3f.slice/crio-7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926 WatchSource:0}: Error finding container 7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926: Status 404 returned error can't find the container with id 7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926
Nov 03 20:48:13.586683 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.586621    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjkp6" event={"ID":"40fb7803-b57b-4822-b933-41bfa1c40f24","Type":"ContainerStarted","Data":"4cb505bfbbc2626150648954fba22c449e8164ac22f3a3044f201b7dad5922f1"}
Nov 03 20:48:13.587185 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.586714    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjkp6" event={"ID":"40fb7803-b57b-4822-b933-41bfa1c40f24","Type":"ContainerStarted","Data":"1efcc87bd3dedeaba3f48205daab15a509ef11ee48c3adacb0a53a7788d535ce"}
Nov 03 20:48:13.588927 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.588894    2567 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da" exitCode=0
Nov 03 20:48:13.589811 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.589702    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"}
Nov 03 20:48:13.589811 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.589777    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerStarted","Data":"7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926"}
Nov 03 20:48:13.697755 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:13.697698    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fjkp6" podStartSLOduration=66.206675291 podStartE2EDuration="1m7.697679623s" totalImagesPullingTime="1.491004332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:13.696780234 +0000 UTC m=+67.840929998" watchObservedRunningTime="2026-11-03 20:48:13.697679623 +0000 UTC m=+67.841829383"
Nov 03 20:48:15.464243 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.464205    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:48:15.469373 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.468902    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.520770 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.520724    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:48:15.568217 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.568179    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:48:15.583412 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583563 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583563 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583563 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583563 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583517    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583563 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583824 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.583824 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.583677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685127    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685356    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.685769 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.685398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.687215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.686176    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.687215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.686281    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.687215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.686514    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.687215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.686817    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.687215 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.686936    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.690500 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.690477    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.702841 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.702815    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.745398 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.745336    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:15.785149 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:15.785122    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:16.164977 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.164776    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:48:16.168362 ip-10-0-140-144 kubenswrapper[2567]: W1103 20:48:16.168297    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714afbba_883e_46b4_927f_473d9d940307.slice/crio-14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81 WatchSource:0}: Error finding container 14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81: Status 404 returned error can't find the container with id 14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81
Nov 03 20:48:16.605739 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.605683    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4tkw" event={"ID":"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39","Type":"ContainerStarted","Data":"82dbd0d37d4ead6ab9cb157d8229b5cb6b51edb6f29ebb951b004f5aba3d2c9c"}
Nov 03 20:48:16.606233 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.605966    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:16.608109 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.608080    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4747674d-vffft" event={"ID":"714afbba-883e-46b4-927f-473d9d940307","Type":"ContainerStarted","Data":"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"}
Nov 03 20:48:16.608263 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.608131    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4747674d-vffft" event={"ID":"714afbba-883e-46b4-927f-473d9d940307","Type":"ContainerStarted","Data":"14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81"}
Nov 03 20:48:16.719628 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.719534    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4tkw" podStartSLOduration=65.307709599 podStartE2EDuration="1m9.719516155s" totalImagesPullingTime="4.411806556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:16.717938791 +0000 UTC m=+70.862088553" watchObservedRunningTime="2026-11-03 20:48:16.719516155 +0000 UTC m=+70.863665919"
Nov 03 20:48:16.831195 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:16.831145    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d4747674d-vffft" podStartSLOduration=1.831130898 podStartE2EDuration="1.831130898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:16.829578777 +0000 UTC m=+70.973728559" watchObservedRunningTime="2026-11-03 20:48:16.831130898 +0000 UTC m=+70.975280659"
Nov 03 20:48:18.303780 ip-10-0-140-144 kubenswrapper[2567]: I1103 20:48:18.303658    2567 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:48:18.303877 ip-10-0-140-144 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:48:18.321076 ip-10-0-140-144 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:48:18.321245 ip-10-0-140-144 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:48:18.321291 ip-10-0-140-144 systemd[1]: kubelet.service: Consumed 3.855s CPU time.
Nov 03 20:48:18.335349 ip-10-0-140-144 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:48:18.397936 ip-10-0-140-144 kubenswrapper[5533]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:48:18.397936 ip-10-0-140-144 kubenswrapper[5533]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:48:18.397936 ip-10-0-140-144 kubenswrapper[5533]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:48:18.397936 ip-10-0-140-144 kubenswrapper[5533]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:48:18.400410 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400335    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.400410 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400407    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.400410 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400410    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.400410 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400413    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.400410 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400416    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400419    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400422    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400425    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400430    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400434    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400438    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400441    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400443    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400446    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400449    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400452    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400454    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400457    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400460    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400462    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400465    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400468    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400470    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.400685 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400473    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400476    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400479    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400482    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400488    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400491    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400495    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400500    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400502    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400505    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400508    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400510    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400513    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400516    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400518    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400521    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400523    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400526    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400529    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.401157 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400531    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400534    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400536    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400539    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400542    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400545    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400548    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400551    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400554    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400557    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400559    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400562    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400564    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400567    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400570    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400572    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400575    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400577    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400580    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400583    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.401615 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400586    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400588    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400591    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400593    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400596    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400598    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400601    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400603    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400606    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400609    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400611    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400614    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400616    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400619    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400621    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400624    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400627    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400630    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400649    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400652    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.402134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400654    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400658    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400660    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400663    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400666    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400669    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400671    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400674    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400676    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400679    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400682    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400685    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400688    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400691    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400694    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400697    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400699    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400702    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400707    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400710    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.402617 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400712    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400716    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400719    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.400722    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401438    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401445    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401448    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401451    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401454    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401457    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401459    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401462    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401465    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401467    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401470    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401472    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401475    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401478    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401480    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401483    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.403134 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401486    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401490    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401492    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401495    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401498    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401501    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401504    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401506    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401509    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401513    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401519    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401523    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401526    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401529    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401532    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401535    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401537    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401540    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401542    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.403624 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401545    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401548    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401550    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401553    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401555    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401558    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401561    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401563    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401566    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401568    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401571    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401573    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401576    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401579    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401582    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401584    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401587    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401589    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401592    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401595    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.404127 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401598    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401600    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401603    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401608    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401611    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401613    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401616    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401619    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401623    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401626    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401628    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401647    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401652    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401655    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401658    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401661    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401663    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401666    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401669    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.404658 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401672    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401675    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401677    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401680    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401682    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401685    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401689    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401692    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401694    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401697    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401700    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401703    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401706    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401709    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401712    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401714    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401719    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401721    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401724    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.405165 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401726    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401729    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401732    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401734    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401737    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401739    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401742    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401745    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401747    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401750    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401752    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401755    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.401758    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402248    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402254    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402258    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402261    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402264    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402267    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.405622 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402270    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402273    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402276    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402279    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402281    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402285    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402288    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402290    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402293    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402296    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402299    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402302    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402305    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402307    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402310    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402312    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402315    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402318    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402321    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402323    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.406161 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402326    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402328    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402332    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402337    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402340    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402343    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402346    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402349    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402352    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402355    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402357    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402361    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402366    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402369    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402371    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402374    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402376    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402379    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402382    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.406696 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402385    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402387    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402390    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402393    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402396    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402399    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402401    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402404    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402407    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402409    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402412    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402414    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402417    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402419    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402422    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402425    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402427    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402430    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402433    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.407173 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402436    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402438    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402441    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402443    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402446    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402449    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402453    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402455    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402458    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402461    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402463    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402466    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402468    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402471    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402474    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402477    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402479    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402482    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402484    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.407651 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402487    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402489    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402492    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402494    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402497    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402499    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402501    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402504    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402507    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402509    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402512    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402514    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402517    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402519    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402522    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402524    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402527    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402529    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402532    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402537    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.408377 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402540    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402542    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.402545    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402660    5533 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402674    5533 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402680    5533 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402684    5533 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402688    5533 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402693    5533 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402697    5533 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402703    5533 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402707    5533 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402710    5533 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402713    5533 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402717    5533 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402720    5533 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402724    5533 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402726    5533 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402730    5533 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402733    5533 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402736    5533 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402741    5533 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402744    5533 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402747    5533 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:48:18.409106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402750    5533 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402754    5533 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402758    5533 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402761    5533 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402764    5533 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402767    5533 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402770    5533 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402775    5533 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402778    5533 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402781    5533 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402784    5533 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402789    5533 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402792    5533 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402795    5533 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402798    5533 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402801    5533 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402804    5533 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402809    5533 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402813    5533 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402817    5533 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402820    5533 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402823    5533 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402827    5533 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402831    5533 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402834    5533 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:48:18.409726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402837    5533 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402840    5533 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402843    5533 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402846    5533 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402849    5533 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402852    5533 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402855    5533 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402858    5533 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402862    5533 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402865    5533 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402868    5533 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402872    5533 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402875    5533 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402878    5533 flags.go:64] FLAG: --help="false"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402882    5533 flags.go:64] FLAG: --hostname-override="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402886    5533 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402889    5533 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402892    5533 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402896    5533 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402899    5533 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402902    5533 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402905    5533 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402908    5533 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:48:18.410350 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402911    5533 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402914    5533 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402919    5533 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402923    5533 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402926    5533 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402928    5533 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402932    5533 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402935    5533 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402938    5533 flags.go:64] FLAG: --lock-file=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402940    5533 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402944    5533 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402947    5533 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402952    5533 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402955    5533 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402958    5533 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402961    5533 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402964    5533 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402967    5533 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402970    5533 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402973    5533 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402978    5533 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402981    5533 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402985    5533 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402989    5533 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402992    5533 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:48:18.410931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402995    5533 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.402998    5533 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403001    5533 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403004    5533 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403007    5533 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403020    5533 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403025    5533 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403028    5533 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403031    5533 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403036    5533 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403040    5533 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403043    5533 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403046    5533 flags.go:64] FLAG: --port="10250"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403049    5533 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403052    5533 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00aa1d9e4dc4bd4a9"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403055    5533 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403058    5533 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403061    5533 flags.go:64] FLAG: --register-node="true"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403064    5533 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403068    5533 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403071    5533 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403074    5533 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403077    5533 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403081    5533 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403084    5533 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403087    5533 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:48:18.411552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403090    5533 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403093    5533 flags.go:64] FLAG: --runonce="false"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403097    5533 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403100    5533 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403104    5533 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403108    5533 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403110    5533 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403114    5533 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403116    5533 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403119    5533 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403122    5533 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403125    5533 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403128    5533 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403131    5533 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403134    5533 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403139    5533 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403142    5533 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403148    5533 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403151    5533 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403154    5533 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403159    5533 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403162    5533 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403165    5533 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403168    5533 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403171    5533 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:48:18.412207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403174    5533 flags.go:64] FLAG: --v="2"
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403179    5533 flags.go:64] FLAG: --version="false"
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403183    5533 flags.go:64] FLAG: --vmodule=""
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403187    5533 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403190    5533 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403345    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403349    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403352    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403355    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403358    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403360    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403364    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403367    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403369    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403372    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403375    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403377    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403380    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403383    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403385    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403387    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.412841 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403390    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403394    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403397    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403399    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403403    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403405    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403408    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403411    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403413    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403416    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403418    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403421    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403424    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403427    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403429    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403432    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403434    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403439    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403442    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.413408 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403446    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403449    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403452    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403454    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403457    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403459    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403462    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403465    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403467    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403470    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403472    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403475    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403478    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403481    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403485    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403487    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403490    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403492    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403496    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403499    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.413962 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403501    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403504    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403507    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403509    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403512    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403515    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403517    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403520    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403523    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403525    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403528    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403530    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403533    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403535    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403538    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403541    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403544    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403546    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403549    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.414676 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403551    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403554    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403557    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403560    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403562    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403565    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403568    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403572    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403574    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403577    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403580    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403582    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403585    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403588    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403591    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403594    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403597    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403599    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403602    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.415164 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403605    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403607    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403610    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403613    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403617    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403620    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403623    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403626    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403629    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403644    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403647    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403649    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403652    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.403678    5533 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403715    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403719    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.415627 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403722    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403725    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403728    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403733    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403737    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403742    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403745    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403748    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403750    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403753    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403756    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403759    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403762    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403765    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403769    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403771    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403774    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403777    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403779    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.416176 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403782    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403785    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403787    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403790    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403793    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403795    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403798    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403801    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403803    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403806    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403808    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403811    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403813    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403816    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403819    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403821    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403824    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403827    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403832    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403834    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.416801 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403837    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403839    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403842    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403845    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403848    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403851    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403853    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403857    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403860    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403863    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403866    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403869    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403871    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403874    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403876    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403879    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403882    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403884    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403887    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.417323 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403890    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403893    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403895    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403898    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403911    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403916    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403920    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403924    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403928    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403933    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403937    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403945    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403949    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403952    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403956    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403960    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403963    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403967    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403973    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403977    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.417809 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403983    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403987    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403991    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403994    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.403998    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404002    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404006    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404010    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404014    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404018    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404022    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404026    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404031    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404034    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404036    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404039    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404042    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404045    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404047    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404050    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.418290 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404053    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404055    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404058    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404062    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404065    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.404067    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.409134    5533 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.409150    5533 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.409177    5533 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.409182    5533 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409242    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409248    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409251    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409254    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409257    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409260    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409263    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409265    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409268    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.419038 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409271    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409274    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409277    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409279    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409282    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409285    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409288    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409291    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409293    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409296    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409298    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409301    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409303    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409306    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409309    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409311    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409314    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409317    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409320    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409322    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.419504 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409325    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409327    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409330    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409332    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409335    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409338    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409340    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409343    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409345    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409348    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409350    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409353    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409356    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409358    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409361    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409364    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409366    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409370    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409373    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409376    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.420023 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409378    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409382    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409386    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409389    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409393    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409396    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409398    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409401    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409404    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409407    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409409    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409412    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409415    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409418    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409420    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409423    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409426    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409428    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409431    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.420505 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409434    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409436    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409439    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409441    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409444    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409447    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409449    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409452    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409454    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409457    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409460    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409463    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409466    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409468    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409471    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409474    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409476    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409479    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409482    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.421095 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409484    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409487    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409490    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409492    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409495    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409497    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409500    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409502    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409505    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409508    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409510    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409513    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409516    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409518    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409522    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409526    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409529    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409532    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.421732 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409535    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.409563    5533 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409719    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409725    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409728    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409731    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409734    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409738    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409741    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409744    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409746    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409749    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409752    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409754    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409757    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409759    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.422227 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409762    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409766    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409769    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409772    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409775    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409778    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409781    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409783    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409788    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409791    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409793    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409796    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409798    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409801    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409804    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409806    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409809    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409812    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409814    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409817    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.422607 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409820    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409822    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409825    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409828    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409832    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409835    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409838    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409840    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409843    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409846    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409848    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409851    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409853    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409856    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409858    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409861    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409864    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409866    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409869    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409871    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.423115 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409876    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409878    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409881    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409884    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409886    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409889    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409892    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409895    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409899    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409902    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409906    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409908    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409911    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409914    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409916    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409919    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409941    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409946    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.423614 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409948    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409951    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409954    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409956    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409959    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409961    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409964    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409966    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409969    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409972    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409974    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409977    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409979    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409982    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409986    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409989    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409992    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409994    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.409997    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410000    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.424345 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410002    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410005    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410007    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410010    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410012    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410015    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410017    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410020    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410023    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410025    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410028    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410030    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410033    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410036    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.410060    5533 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410098    5533 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:48:18.425107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410102    5533 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410105    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410108    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410111    5533 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410126    5533 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410130    5533 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410135    5533 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410138    5533 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410141    5533 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410144    5533 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410148    5533 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410151    5533 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410153    5533 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410156    5533 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410159    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410162    5533 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410165    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410167    5533 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410170    5533 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410173    5533 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:48:18.425543 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410175    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410178    5533 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410181    5533 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410183    5533 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410186    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410189    5533 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410191    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410194    5533 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410197    5533 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410201    5533 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410204    5533 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410206    5533 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410209    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410211    5533 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410214    5533 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410218    5533 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410222    5533 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410226    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410230    5533 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:48:18.426044 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410233    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410235    5533 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410238    5533 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410242    5533 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410244    5533 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410247    5533 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410250    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410252    5533 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410255    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410257    5533 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410260    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410263    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410265    5533 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410268    5533 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410270    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410273    5533 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410276    5533 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410278    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410281    5533 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:48:18.426526 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410284    5533 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410286    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410289    5533 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410291    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410294    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410297    5533 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410299    5533 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410302    5533 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410305    5533 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410307    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410309    5533 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410312    5533 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410316    5533 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410319    5533 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410321    5533 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410324    5533 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410328    5533 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410331    5533 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410334    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410336    5533 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:48:18.427060 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410339    5533 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410341    5533 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410345    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410347    5533 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410351    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410353    5533 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410356    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410359    5533 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410362    5533 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410365    5533 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410368    5533 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410370    5533 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410373    5533 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410376    5533 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410378    5533 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410381    5533 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410383    5533 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410388    5533 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410391    5533 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410409    5533 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:48:18.427562 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410412    5533 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410415    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410418    5533 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410421    5533 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410426    5533 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410428    5533 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:48:18.410431    5533 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.410483    5533 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411406    5533 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411455    5533 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411622    5533 server.go:1014] "Starting client certificate rotation"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411737    5533 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411779    5533 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:42:06 +0000 UTC" deadline="2028-08-21 03:22:51.265689862 +0000 UTC"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.411814    5533 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15750h34m32.853877513s"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.412098    5533 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.413123    5533 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:48:18.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.414152    5533 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:48:18.428452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.415981    5533 log.go:25] "Validated CRI v1 image API"
Nov 03 20:48:18.428452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.416746    5533 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:48:18.428452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.418626    5533 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad0b69a6-16ee-44ee-97ca-ec4ecb5df7a0:/dev/nvme0n1p3 cffd87f9-d003-4e32-9fed-649586e51184:/dev/nvme0n1p4]
Nov 03 20:48:18.428558 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.418663    5533 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbc7e3f16a09cc9e24d6799d6497e61a5c52f19465143e2298e487bac545665e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbc7e3f16a09cc9e24d6799d6497e61a5c52f19465143e2298e487bac545665e/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/alertmanager/9 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/config-reloader/6 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/kube-api-access-rw5wf:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/kube-api-access-rw5wf major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/config-volume major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/web-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c429b7-91ff-41b2-96fc-575e8cc12640/volumes/kubernetes.io~projected/kube-api-access-flvpg:{mountpoint:/var/lib/kubelet/pods/13c429b7-91ff-41b2-96fc-575e8cc12640/volumes/kubernetes.io~projected/kube-api-access-flvpg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33fd3340-3104-450d-a434-0e89d73d7331/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/33fd3340-3104-450d-a434-0e89d73d7331/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~projected/kube-api-access-npbvh:{mountpoint:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~projected/kube-api-access-npbvh major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~projected/kube-api-access-vhzzc:{mountpoint:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~projected/kube-api-access-vhzzc major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~projected/kube-api-access-j82gp:{mountpoint:/var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~projected/kube-api-access-j82gp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~secret/metrics-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~projected/kube-api-access-28r6x:{mountpoint:/var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~projected/kube-api-access-28r6x major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6418ffe1-921f-4045-9e38-761687df3ad7/volumes/kubernetes.io~projected/kube-api-access-mfs54:{mountpoint:/var/lib/kubelet/pods/6418ffe1-921f-4045-9e38-761687df3ad7/volumes/kubernetes.io~projected/kube-api-access-mfs54 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~projected/kube-api-access-kddtk:{mountpoint:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~projected/kube-api-access-kddtk major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~projected/kube-api-access-dzz4b:{mountpoint:/var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~projected/kube-api-access-dzz4b major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b1c9b27-49d4-4d22-9ff7-350c94e96665/volumes/kubernetes.io~projected/kube-api-access-cshvv:{mountpoint:/var/lib/kubelet/pods/8b1c9b27-49d4-4d22-9ff7-350c94e96665/volumes/kubernetes.io~projected/kube-api-access-cshvv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/prometheus/15 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/kube-api-access-z2xfh:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/kube-api-access-z2xfh major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/tls-assets major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/config major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/web-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/983d3627-42d7-4192-ba09-c44043b657ee/volumes/kubernetes.io~projected/kube-api-access-zsb79:{mountpoint:/var/lib/kubelet/pods/983d3627-42d7-4192-ba09-c44043b657ee/volumes/kubernetes.io~projected/kube-api-access-zsb79 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~projected/kube-api-access-6kf96:{mountpoint:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~projected/kube-api-access-6kf96 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb57a79e-5114-4c7f-97f5-23b45aebd6e7/volumes/kubernetes.io~projected/kube-api-access-sz4nr:{mountpoint:/var/lib/kubelet/pods/bb57a79e-5114-4c7f-97f5-23b45aebd6e7/volumes/kubernetes.io~projected/kube-api-access-sz4nr major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39/volumes/kubernetes.io~projected/kube-api-access-h2p9g:{mountpoint:/var/lib/kubelet/pods/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39/volumes/kubernetes.io~projected/kube-api-access-h2p9g major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~projected/kube-api-access-6sgjk:{mountpoint:/var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~projected/kube-api-access-6sgjk major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~projected/kube-api-access-v5g6d:{mountpoint:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~projected/kube-api-access-v5g6d major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb59a160-8501-4557-b55e-e4f3e87cfa61/volumes/kubernetes.io~projected/kube-api-access-l69x2:{mountpoint:/var/lib/kubelet/pods/eb59a160-8501-4557-b55e-e4f3e87cfa61/volumes/kubernetes.io~projected/kube-api-access-l69x2 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~projected/kube-api-access-tzp26:{mountpoint:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~projected/kube-api-access-tzp26 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:230 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/66d669306749ae0482745afb3d234e9d4fb1a4e2a96c3644b40664af5a3cc4d3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4ea44470cf5d20c45a4ae807825d3829a851bcb1de72aca24a12ca344cee9f53/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f3148a8dc4e9de436d6222ff3c7eef5c15391682e3d96ab9b7b7443625a689ad/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/96f27fd4572222aea79b6be70bb5337e71e51931fa7faa5e95e5fcfa3b3739fa/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/46846c29c2b7184699c23c11dea7800e69880595607e81bed3747b7e4a649153/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6747419d3f10a4e4c2279b7d6f3a283cec25b6882ff12bed0b7b8501cb3ffc43/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8ac452038f427a6cbe5127de6ea8c1955ca9a62cabb3b7dc31e5e86b608b6538/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a59f061b0ed2560bdf7ef619af4309d3cf98032f60455aad48e36423e60373fe/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d8962f855f15bd325ad66b8d7dfe27ec166b0a29f17625a413677f86fb1336c6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/96889e2a331888af63d172f651be99f9726c3b5aac3bb1290a04ceac0faf7b10/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/34dd33878fe19efda96975ac36dd8e5d6790e584ac43b62768f9fbf61a2fbd09/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f546c3a3b7f7f9d1c14b1fefa06c164d72389e520d057810f32d4ccdd6f8aa12/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/24a9f54799d778bcc57beaeef2431564871444e7269851e7e09f7c0b054b6b78/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d5c47ce0ed174ad7cb9761c3e0ca42949979a0e198557b730d435c6327be1c91/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1100e867443c857087dc6791829490cececf4bd4e661020db2ba24c4d8d8060b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/080e18de96fbbc5ee9f0e7c34e40aaa88616ef6c904897ffa29d5153947f5334/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/80cfacb5e9f3c6c48dc202f9113a104e9bb0fff72cdeaa908a200cbab1147d8b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f243555474b7d2923035e4ce032ec8891d452ed7fe6c6f832d6fcb2f894eee48/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0a9310552e63b4af5cb25e23cda3ef649ee611e58982229a45dc798ae867020c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cb587e8e2a8a5cc9b3b8da4dfab559d687cb4e1a6962b81c351dbd86006d7571/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6c10029d288a63f7f31d35c00ce182bc89252353d032b20ed6046c0d0d9bf4f1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3e9b8fc57b4e106d6bc32b11f9d3b1607e8c3e2f1d3edf15a16ff7c929643b41/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b8922f9a2ca4778387fc10e07fe6fa6b902496168cb48bb96706ae1352f84832/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a1242e0b1300e558c6865470b676f01b34a666d92a845a1ea7e8044946283d4d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/450cbabe48448aeb9c391a321f7acab8cf938ac16f69f92a58f43128d71eef2c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c28f98412c31048261ea6e9af79b5b8803a1098b5ee61ef222882cc244dfee63/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/73e16bb32b74b9c507a2209755241b1ed0cf6beac2f2d01c60e1e12fd3e25e24/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/10d012ccf1aa9f7aa181a1e043904f93a74d5a10b336944b7c84755a938b1404/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/38e72de66bd265333848162cb9f6576d74b30ec4427dc859c9cec62ff8c25c38/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/8531a205e6ff54ba032f40e952f2d0c452347456537d2036f79524a7db69b16b/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/2be747bcea058590273150333a198c9a2607ae44489c56afa703e257265dfe0c/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/d5e09f9c0229d2a18352a45f16aa7b46b6d666643da0bd5109ef3f7eabe964d5/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/59c6b17785ecf71ba08d5351434545a342decd4748107ab401c44be690bb2cb7/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/f8193cb51f921cd938812cc8b3b785f38de0c40e70c82866c6b299cb705fcf6c/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a50c7ed0b42ada0b767b58475ccbf3d1fd07172a7d1b594f9a369e78e443f018/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/f531815ab201c62f2c69853e0ae57813cfc00edcb9d1bc9cffe5c3d783960d7d/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/00f7d449ce45a7365c01e541a5a0546be740fffce6e062209340d7d11320a310/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/66a2df0310907e22387d21362021d470606fce35958b76f1b40a122f7a7c61b1/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d069d5388afc50335e399a16801c8d82fc8b42f384f3d58f114fdb74939a38ed/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/512c018243278f87ac92a70dbfeb503eff9e034a8318882a8038eab6ff11f468/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/5d463e1dc5307f8d889e1b2cef39b93fb38353607c4214f50afd77f1a9627856/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/22174e0185fac2a4f4c89ced1f4a2c62a370fff76052d48c70cdca244526f39f/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/2e00242056f1512dd56dc06734a1e2b1de7208ad39e72604ba58b26a78f2c072/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/1da40ae0aa1e86b0fa611966f728477009f5394c8a227aae3816cfac0553d56b/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/a9e45c67bb92c6933399828c382a34b93fb9130fcfacf572d2bba9e2b534084d/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/1f93ebd73eb9b57de01fea24038ce61104d375fd5b79641b5bd38e34a5de32f1/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/91e1d51f28dfd12d702a02a81848784f14695e5b4332b587c0d7140e7135bd35/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/f765817191f80ca1861fc54e1b5cd593abfdc3578a3dda4847516bc8e509720d/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/95a1293e1283883bacd840729e0c321145a6d9a1d1e6a9e64226cd09d16e6e93/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/bc8a32f9a3cfda1b69a64859f20474408a01ea17a3362383ffb6656f130c32db/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/dba6b9e5a5748aff1b8026683b350f0f6cb7181f137d2c839ced848397eff903/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/5e9e155320911fde86fa6320008e47cc34155a994666e5ef8e4b6359dfb1c798/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/9b3af6de80d9a1b4011ee5bf132189dbc0045b682a826a3842053cf0732e58fb/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/ea2072c039ab18d51032c52662d3fa0b4d0edf30ae6aa0505d5f53af0e19afcb/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/9c802691ff891f9db52f8664766aff8cfc83a744c54f29e6852e8c1887b0ce8b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/b513bace8c160c8253dce35c646edacc16f08369ab8fd06c93e61fadf8004955/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/2a340b3c3fe6ebb1b23d5d92dca712a6f679c9c2ba6bbe8f0d55520ab7e51dbf/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/b2744ddfa48708a831b455e20af8051ff0705f0a7b670ca45a31f137abf265cf/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f6a0ab168e78662deb3e94b996fe5c6712d76cb7d8fff610fbcb74bddfbaac9a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/bd539fc09ec058e8fdf885f026e956d39416193cff6fb5afa485602644cf64d0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3b2e6f937ce4445b767252602812ef7a8bbd2cb8363eca8ff1da413bb5acad76/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f061809d7c9f3bdad80e157ee1201df8fd7104b324b389bd24d7ede4c7936f5b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/54f1dd28efd66e8b7c44a351f0080b00fa99da778deb09f7d403005eb16ba808/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b95215b580a41202cb704a84b537826f39fa0e52c0d58741e5ee505f15b797c7/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3ed7117378e7b8bc9ad3564c02a8518c2269fd7742869b5b80d6d93fb1458b62/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/405d3f7224f4c807cef6ff4abc15fe93b9d52c92f67d7851106b683a623f5275/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/149adef04415fd90b0681d7c8b5df9d4a824e08d84d871a884b54451386815f6/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.427690    5533 manager.go:217] Machine: {Timestamp:2026-11-03 20:48:18.427213325 +0000 UTC m=+0.058526793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099806 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2493f0b6a5c3c1512990392abce7ac SystemUUID:ec2493f0-b6a5-c3c1-5129-90392abce7ac BootID:f2de8a4b-0254-47ea-aa44-75260c952258 Filesystems:[{Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b11a41ca32a84e057bc4f0effae366e823d8301da1038d09a556985ec5762c9/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b8079346a720b6263cf7bbf31effcda23401b50e770fbb63852abb5248a442d/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb57a79e-5114-4c7f-97f5-23b45aebd6e7/volumes/kubernetes.io~projected/kube-api-access-sz4nr DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~projected/kube-api-access-kddtk DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3504db39de95822f8c8e2d23b1db900432235f58dcb0e87cbb0e11feb2b8c31/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7e12adfb80fddbcbb30c657f62595dde59491595e5f273bc62124dbf4ccf700/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:366 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~projected/kube-api-access-vhzzc DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6b3a4dbe68c7cb16e6377344b99df737176e0ca0b2fd959a9af619955112162/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb59a160-8501-4557-b55e-e4f3e87cfa61/volumes/kubernetes.io~projected/kube-api-access-l69x2 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13c429b7-91ff-41b2-96fc-575e8cc12640/volumes/kubernetes.io~projected/kube-api-access-flvpg DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbc7e3f16a09cc9e24d6799d6497e61a5c52f19465143e2298e487bac545665e/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7d8677f-d924-45ce-8ff2-75fe9725346c/volumes/kubernetes.io~projected/kube-api-access-6sgjk DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fac07d09aba0563b543b5855db8824161680cdafc7538d252eb73541e6bbab2a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6418ffe1-921f-4045-9e38-761687df3ad7/volumes/kubernetes.io~projected/kube-api-access-mfs54 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d775119c6166d9830b4548f74b1010844ef8f65782c74a6de2949dfceebdc24/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56c813be-5ca7-4418-891f-04939346e716/volumes/kubernetes.io~projected/kube-api-access-28r6x DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa7d271595be7baa7ff4744603e2968c015447ba28a323e06da4436ea12974a3/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39/volumes/kubernetes.io~projected/kube-api-access-h2p9g DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb7803-b57b-4822-b933-41bfa1c40f24/volumes/kubernetes.io~projected/kube-api-access-j82gp DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72ed3e0e3aab889550ce5c36ccf3fc7ff322962e8955608b4db84e387c953611/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/volumes/kubernetes.io~projected/kube-api-access-6kf96 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/686f1a2a0d57b5323ea15bb2dd4c47bcde8b1abae06561f29f3863feaadd6d8f/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37651c6c-a9ce-4c30-af0f-7e74869a7ef4/volumes/kubernetes.io~projected/kube-api-access-npbvh DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/067ecbd0183be889c266e025252ecb32caa4ae385664e421505a2c578c00966b/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b715b9c48cd35d606669668fae7a343ea46153c4b3bd7f3cfbace2d9e185f33/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8fc55c3-89a7-4d55-bae4-a64d15efebaa/volumes/kubernetes.io~projected/kube-api-access-v5g6d DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7eda94b8-a813-448e-8d29-742cd73e5d80/volumes/kubernetes.io~projected/kube-api-access-dzz4b DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86f5b68735b25d1f61b636aa84d4a10b4f6a1553738fe6b79ff4b9209bd1abe9/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33fd3340-3104-450d-a434-0e89d73d7331/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64a65725604f723338a29ee673e80dfc4f20b7074bf221de809ed220548fde17/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes/kubernetes.io~projected/kube-api-access-z2xfh DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3476f13dc608a9310c884a4577e1cfc2f6ed8f495854451ae897cc6b848cda83/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes/kubernetes.io~projected/kube-api-access-tzp26 DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes/kubernetes.io~projected/kube-api-access-rw5wf DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/983d3627-42d7-4192-ba09-c44043b657ee/volumes/kubernetes.io~projected/kube-api-access-zsb79 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa08991-01cf-45cd-a970-0dba2a38a851/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b1c9b27-49d4-4d22-9ff7-350c94e96665/volumes/kubernetes.io~projected/kube-api-access-cshvv DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14e9e58bf0b2324 MacAddress:0e:91:fe:3a:8e:62 Speed:10000 Mtu:8901} {Name:2b715b9c48cd35d MacAddress:42:51:c1:16:1f:6a Speed:10000 Mtu:8901} {Name:2fe9a01eb14b11f MacAddress:56:61:62:1e:a5:2a Speed:10000 Mtu:8901} {Name:64a65725604f723 MacAddress:26:53:fb:e8:08:e1 Speed:10000 Mtu:8901} {Name:686f1a2a0d57b53 MacAddress:92:76:a9:88:a3:0a Speed:10000 Mtu:8901} {Name:7f801c0cff104a4 MacAddress:86:c2:93:46:c4:ef Speed:10000 Mtu:8901} {Name:86f5b68735b25d1 MacAddress:c6:3b:29:bc:56:46 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d0:9c:16:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:7d:05:93:8b:5e Speed:0 Mtu:8901} {Name:c72a604e6f2d276 MacAddress:06:d5:20:13:05:5a Speed:10000 Mtu:8901} {Name:dbc7e3f16a09cc9 MacAddress:aa:3a:4f:45:c8:10 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d0:9c:16:f1 Speed:0 Mtu:9001} {Name:fa7d271595be7ba MacAddress:76:68:b2:34:64:a2 Speed:10000 Mtu:8901} {Name:fac07d09aba0563 MacAddress:12:4b:45:c4:9f:26 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:26:ba:d3:a6:7b:3f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:11:8d:44:28:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428086    5533 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428128    5533 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428463    5533 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428486    5533 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-144.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428606    5533 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428613    5533 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428627    5533 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428656    5533 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:48:18.428861 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428728    5533 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428887    5533 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.428982    5533 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.429013    5533 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.429021    5533 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.429031    5533 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.429039    5533 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:48:18.429400 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.429046    5533 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:48:18.430039 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.430023    5533 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:48:18.430039 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.430036    5533 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:48:18.430458 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.430434    5533 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:48:18.443712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.443693    5533 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:48:18.443712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.443707    5533 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:48:18.443824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.443814    5533 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:48:18.444056 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444042    5533 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:48:18.444093 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444073    5533 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:48:18.444192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444178    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:48:18.444192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444193    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444199    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444206    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444214    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444225    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444233    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444239    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444246    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444252    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444260    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444269    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444293    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:48:18.444313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.444302    5533 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:48:18.447648 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.447621    5533 server.go:1263] "Started kubelet"
Nov 03 20:48:18.447752 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.447723    5533 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:48:18.447820 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.447756    5533 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:48:18.447866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.447839    5533 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:48:18.448245 ip-10-0-140-144 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:48:18.448734 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.448295    5533 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.454815    5533 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.455052    5533 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.455065    5533 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.455179    5533 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.455324    5533 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:48:18.455678 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.455598    5533 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:48:18.456840 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.456780    5533 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:42:07 +0000 UTC" deadline="2028-08-01 21:14:41.862592012 +0000 UTC"
Nov 03 20:48:18.456840 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.456839    5533 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15288h26m23.405756996s"
Nov 03 20:48:18.458214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.456948    5533 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:48:18.458214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458186    5533 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:48:18.458214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458198    5533 factory.go:55] Registering systemd factory
Nov 03 20:48:18.458214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458205    5533 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:48:18.458417 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458401    5533 factory.go:153] Registering CRI-O factory
Nov 03 20:48:18.458417 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458419    5533 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:48:18.458511 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458432    5533 factory.go:103] Registering Raw factory
Nov 03 20:48:18.458511 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458442    5533 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:48:18.458843 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.458826    5533 manager.go:319] Starting recovery of all containers
Nov 03 20:48:18.459622 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459541    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459650    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec" volumeName="kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459666    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33fd3340-3104-450d-a434-0e89d73d7331" volumeName="kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459680    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb7803-b57b-4822-b933-41bfa1c40f24" volumeName="kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459693    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459706    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 20:48:18.459742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459719    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459747    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" volumeName="kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459776    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459802    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459829    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37651c6c-a9ce-4c30-af0f-7e74869a7ef4" volumeName="kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459843    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37651c6c-a9ce-4c30-af0f-7e74869a7ef4" volumeName="kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459855    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459880    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983d3627-42d7-4192-ba09-c44043b657ee" volumeName="kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459898    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" volumeName="kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459913    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459928    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6418ffe1-921f-4045-9e38-761687df3ad7" volumeName="kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459968    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459981    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert" seLinuxMountContext=""
Nov 03 20:48:18.460018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.459994    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460006    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460040    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460069    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb57a79e-5114-4c7f-97f5-23b45aebd6e7" volumeName="kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460082    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39" volumeName="kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460094    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460117    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460130    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460143    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460204    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c813be-5ca7-4418-891f-04939346e716" volumeName="kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460218    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6418ffe1-921f-4045-9e38-761687df3ad7" volumeName="kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460230    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eda94b8-a813-448e-8d29-742cd73e5d80" volumeName="kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460261    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460274    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460286    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460310    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460323    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1c9b27-49d4-4d22-9ff7-350c94e96665" volumeName="kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460334    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460357    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460370    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460383    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d8677f-d924-45ce-8ff2-75fe9725346c" volumeName="kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460409    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460422    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460458    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c429b7-91ff-41b2-96fc-575e8cc12640" volumeName="kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460482    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa08991-01cf-45cd-a970-0dba2a38a851" volumeName="kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460518    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1c9b27-49d4-4d22-9ff7-350c94e96665" volumeName="kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460552    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb57a79e-5114-4c7f-97f5-23b45aebd6e7" volumeName="kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460576    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" volumeName="kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460588    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca" seLinuxMountContext=""
Nov 03 20:48:18.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460601    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460624    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460678    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d8677f-d924-45ce-8ff2-75fe9725346c" volumeName="kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460705    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460730    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460754    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460766    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c813be-5ca7-4418-891f-04939346e716" volumeName="kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460789    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c813be-5ca7-4418-891f-04939346e716" volumeName="kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460840    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6418ffe1-921f-4045-9e38-761687df3ad7" volumeName="kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460851    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460880    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb59a160-8501-4557-b55e-e4f3e87cfa61" volumeName="kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460890    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb59a160-8501-4557-b55e-e4f3e87cfa61" volumeName="kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460902    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460914    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33fd3340-3104-450d-a434-0e89d73d7331" volumeName="kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460926    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa08991-01cf-45cd-a970-0dba2a38a851" volumeName="kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460937    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460949    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" volumeName="kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460960    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460980    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460987    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.460994    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983d3627-42d7-4192-ba09-c44043b657ee" volumeName="kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461001    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461008    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6418ffe1-921f-4045-9e38-761687df3ad7" volumeName="kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461019    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461033    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eda94b8-a813-448e-8d29-742cd73e5d80" volumeName="kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461044    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1c9b27-49d4-4d22-9ff7-350c94e96665" volumeName="kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461056    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461066    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461073    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" volumeName="kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461080    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c813be-5ca7-4418-891f-04939346e716" volumeName="kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461087    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461095    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37651c6c-a9ce-4c30-af0f-7e74869a7ef4" volumeName="kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461105    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb7803-b57b-4822-b933-41bfa1c40f24" volumeName="kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461116    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461129    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461136    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461144    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461153    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa08991-01cf-45cd-a970-0dba2a38a851" volumeName="kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461159    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461169    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d8677f-d924-45ce-8ff2-75fe9725346c" volumeName="kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461180    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d8677f-d924-45ce-8ff2-75fe9725346c" volumeName="kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461191    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" volumeName="kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461204    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b31527-43de-413d-8b90-962283a47555" volumeName="kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461216    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461228    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37651c6c-a9ce-4c30-af0f-7e74869a7ef4" volumeName="kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461241    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e14bc-5d36-4152-8049-585724259f3f" volumeName="kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461255    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1195581f-22f9-447f-8f94-c3a08eaa48a1" volumeName="kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461267    5533 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714afbba-883e-46b4-927f-473d9d940307" volumeName="kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config" seLinuxMountContext=""
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461283    5533 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:48:18.461879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.461290    5533 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:48:18.479569 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.479546    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:48:18.483040 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.483009    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:48:18.484165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.484113    5533 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:48:18.485688 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.485594    5533 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:48:18.485688 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.485615    5533 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:48:18.485688 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.485656    5533 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:48:18.485860 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.485702    5533 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:48:18.496406 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.496384    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da" exitCode=0
Nov 03 20:48:18.509169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.509150    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:48:18.509503 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.509483    5533 generic.go:356] "Generic (PLEG): container finished" podID="c8fc55c3-89a7-4d55-bae4-a64d15efebaa" containerID="321f4fb7f31cc513196a80e396dd21163d5fe0ac25e985ac8a6a2a4d66cbe56f" exitCode=1
Nov 03 20:48:18.522794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.522766    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="85420996b2d1fef7e51fd505237eda959587a19601509d4abae8f6f595307878" exitCode=0
Nov 03 20:48:18.531484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.531458    5533 generic.go:356] "Generic (PLEG): container finished" podID="b8dfb03f-f653-43e7-9a8a-4a61bbb2220a" containerID="ddd6161d5314e3c07ffba4227faf477bcd2e7d1b5cd1e21799cf0a8941902463" exitCode=0
Nov 03 20:48:18.540978 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.540957    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="a1050e300d85ba05e31e6b42051a8b2f1aff49b258a6ed5fd96962bfd87882cf" exitCode=0
Nov 03 20:48:18.540978 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.540978    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="f99679f76294099b7321a639d20edc8beb9d85529fba9753bae782edbcc405bf" exitCode=0
Nov 03 20:48:18.541094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.540985    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="bcce2f274f153f4d3cea3f363c9e76e5ca1484fa6e0f7ff4eacd52da6bd96c4d" exitCode=0
Nov 03 20:48:18.541094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.540991    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="f4a1fab13231c7735063ff5020b6db68cef2776228d568598898493484d249b0" exitCode=0
Nov 03 20:48:18.541094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.540997    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="42598fd81d52ca02fed314fe3570497776f446601572137ad51d045cb293f0b8" exitCode=0
Nov 03 20:48:18.541094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.541005    5533 generic.go:356] "Generic (PLEG): container finished" podID="6418ffe1-921f-4045-9e38-761687df3ad7" containerID="7c3a6261e7061f0d3d2f1c2a4f099c538385ec976fd1b4a3251907f232423810" exitCode=0
Nov 03 20:48:18.545794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.545773    5533 generic.go:356] "Generic (PLEG): container finished" podID="1d51c0bbdc067fa97bdab078f3d2d6ac" containerID="2d41d947ce373a785aabb13bf1f0ab9be91c046c968b53aaf70bd01699928bdb" exitCode=0
Nov 03 20:48:18.556070 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.555964    5533 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:48:18.564869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.564854    5533 manager.go:324] Recovery completed
Nov 03 20:48:18.585782 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.585752    5533 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 20:48:18.586789 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.586775    5533 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:48:18.588818 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.588802    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:48:18.588880 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.588831    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:48:18.588880 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.588843    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:48:18.591496 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591481    5533 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:48:18.591496 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591490    5533 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:48:18.591496 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591494    5533 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:48:18.591726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591514    5533 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:48:18.591726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591677    5533 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:48:18.591726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591694    5533 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:48:18.591726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591719    5533 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:48:18.591726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591727    5533 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591736    5533 policy_none.go:50] "Start"
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591748    5533 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591761    5533 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591936    5533 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591948    5533 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:48:18.591970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.591956    5533 policy_none.go:44] "Start"
Nov 03 20:48:18.592376 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.592359    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:48:18.592439 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.592391    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:48:18.592439 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.592406    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:48:18.596762 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.596736    5533 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:48:18.599805 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.599792    5533 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:48:18.599890 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.599821    5533 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:48:18.599890 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.599833    5533 server.go:87] "Starting device plugin registration server"
Nov 03 20:48:18.600119 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600106    5533 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:48:18.600186 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600119    5533 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:48:18.600259 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600243    5533 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:48:18.600389 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600347    5533 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:48:18.600452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600391    5533 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:48:18.600452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600397    5533 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:48:18.601255 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.600446    5533 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:48:18.600376977Z","UUID":"70cc3f2c-fed4-4a80-9226-2e11ddf5faf3","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:48:18.601941 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.601925    5533 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:48:18.602064 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.602050    5533 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-144.us-west-2.compute.internal\" not found"
Nov 03 20:48:18.602488 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.602473    5533 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:48:18.602598 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.602585    5533 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:48:18.642274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.642245    5533 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:48:18.642274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.642245    5533 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:48:18.642417 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.642403    5533 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:48:18.643238 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.643222    5533 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:48:18.700367 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.700346    5533 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:48:18.701210 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.701195    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:48:18.701267 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.701223    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:48:18.701267 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.701233    5533 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:48:18.703251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.701506    5533 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.735262 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.735239    5533 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:48:18.786608 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.786572    5533 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"]
Nov 03 20:48:18.787181 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787166    5533 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.787251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787171    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"8b940c9766e72993f1a0cb04738b1af3","Type":"ContainerStarted","Data":"e5d423321872442a89e991e6d835a2b233c8f94a4db514b53a926c5fab3374e5"}
Nov 03 20:48:18.787315 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787275    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"8b940c9766e72993f1a0cb04738b1af3","Type":"ContainerStarted","Data":"e580063d296363d1c15c6faad7423eb9be3288fb0244054c6afe7be8b5775378"}
Nov 03 20:48:18.787368 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787318    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerStarted","Data":"b059236f4c6caef6ac18f185dd40b5733291de82603efbe20cceab9903866e99"}
Nov 03 20:48:18.787368 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787330    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerDied","Data":"2d41d947ce373a785aabb13bf1f0ab9be91c046c968b53aaf70bd01699928bdb"}
Nov 03 20:48:18.787368 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787350    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal" event={"ID":"1d51c0bbdc067fa97bdab078f3d2d6ac","Type":"ContainerStarted","Data":"22b5a47f7a102d7dc442cbd5043aa1f68bd27688fbe5413be24e5d1d9ed7e0e3"}
Nov 03 20:48:18.787459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.787434    5533 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.806557 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.806507    5533 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.806665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.806561    5533 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.846372 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.846350    5533 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:48:18.846451 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.846389    5533 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.846888 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.846873    5533 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:48:18.846921 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:18.846906    5533 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.944806 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.944765    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.944875 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.944811    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:18.944875 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:18.944827    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045774 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045746    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045856 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045801    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045856 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045835    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045939 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045862    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045939 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045898    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b940c9766e72993f1a0cb04738b1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"8b940c9766e72993f1a0cb04738b1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.045939 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.045915    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d51c0bbdc067fa97bdab078f3d2d6ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" (UID: \"1d51c0bbdc067fa97bdab078f3d2d6ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.430011 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.429976    5533 apiserver.go:52] "Watching apiserver"
Nov 03 20:48:19.528701 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.528668    5533 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:48:19.529365 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.529344    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-n99n6","openshift-monitoring/node-exporter-6ssfh","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-vdkpg","openshift-monitoring/alertmanager-main-0","openshift-console/console-86f484b7d7-ptsf7","openshift-ingress-canary/ingress-canary-bznhk","openshift-insights/insights-runtime-extractor-mv9sd","openshift-multus/network-metrics-daemon-fjkp6","openshift-network-diagnostics/network-check-target-h4tkw","openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9","openshift-multus/multus-8kvc5","kube-system/konnectivity-agent-qj2ct","kube-system/global-pull-secret-syncer-stvfh","openshift-cluster-node-tuning-operator/tuned-9thx7","openshift-dns/dns-default-hjll2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn","openshift-dns/node-resolver-cwmvz","openshift-image-registry/node-ca-mwqr2","kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-s7hmx","openshift-console/console-7d4747674d-vffft"]
Nov 03 20:48:19.550931 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.550911    5533 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.551064 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.551050    5533 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.558075 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.558051    5533 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:48:19.582872 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582854    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.582872 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582866    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582892    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582893    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582927    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582937    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.582954    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.583048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.583005    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.584264 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584247    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.584353 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584303    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:48:19.584353 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584318    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.584353 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584321    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.584484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584464    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.584590 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584577    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.584691 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584678    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:48:19.584749 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584724    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.584749 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584727    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.584844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584727    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:48:19.584844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584758    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.584844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584741    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:48:19.584844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584794    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.584844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584835    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.585022 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584883    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.585022 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.584973    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:48:19.586725 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.586708    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:48:19.586778 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.586726    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:48:19.587119 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.587106    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:48:19.587155 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.587140    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.587197 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.587185    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.600377 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.600359    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:48:19.600583 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.600564    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-868xx\""
Nov 03 20:48:19.600660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.600570    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:48:19.600707 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.600681    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l9gsz\""
Nov 03 20:48:19.600745 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.600736    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldspv\""
Nov 03 20:48:19.601041 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601024    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xldt8\""
Nov 03 20:48:19.601041 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601034    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601046    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7vv6z\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601105    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vwxfp\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601115    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68sfv\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601126    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601120    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6str\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601136    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2g8j2\""
Nov 03 20:48:19.601148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601129    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxhpl\""
Nov 03 20:48:19.601484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601162    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-78pt4\""
Nov 03 20:48:19.601484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601168    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tlf9q\""
Nov 03 20:48:19.601484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601170    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-snm99\""
Nov 03 20:48:19.601484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601284    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:48:19.601670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601604    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gf6gt\""
Nov 03 20:48:19.601670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.601659    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:48:19.637237 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.637219    5533 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:48:19.637303 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:19.637258    5533 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.637585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.637568    5533 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:48:19.637654 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:19.637611    5533 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.us-west-2.compute.internal"
Nov 03 20:48:19.647866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.647846    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.647953 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.647874    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.647953 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.647891    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.647953 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.647905    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.648117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.647962    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.648117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648000    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.648117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648029    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.648117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648064    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.648312 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648113    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzzc\" (UniqueName: \"kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.648312 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648142    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-tuned\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.648312 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648170    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.648312 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648269    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.648451 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648323    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs54\" (UniqueName: \"kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.648451 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648367    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.648451 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648406    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648458    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648477    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82gp\" (UniqueName: \"kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648500    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kf96\" (UniqueName: \"kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648517    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648533    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.648586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648562    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648627    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983d3627-42d7-4192-ba09-c44043b657ee-tmp-dir\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648662    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648689    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648690    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-iptables-alerter-script\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648735    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgjk\" (UniqueName: \"kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648759    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648782    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648805    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648827    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648844    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648854    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648879    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5g6d\" (UniqueName: \"kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648907    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648880    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648933    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648964    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.648965    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cni-binary-copy\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649039    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649084    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649109    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649126    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649143    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649167    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649243    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.649274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649262    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649286    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649326    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649333    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649357    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbvh\" (UniqueName: \"kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649389    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4nr\" (UniqueName: \"kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649412    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649440    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649465    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649489    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649513    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649606    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649628    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvpg\" (UniqueName: \"kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649669    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.649700 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649706    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649722    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649738    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649857    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649927    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649970    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.649997    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650015    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650034    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650086    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.650142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650110    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650146    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650202    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650205    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650251    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d8677f-d924-45ce-8ff2-75fe9725346c-data-volume\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650260    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650267    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40fb7803-b57b-4822-b933-41bfa1c40f24-metrics-certs\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650327    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650368    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650413    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r6x\" (UniqueName: \"kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650455    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650423    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-config\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.650486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650489    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650529    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-daemon-config\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650549    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650578    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650602    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650662    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650687    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650702    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa08991-01cf-45cd-a970-0dba2a38a851-tmp\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650704    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650701    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb59a160-8501-4557-b55e-e4f3e87cfa61-serviceca\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650747    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6418ffe1-921f-4045-9e38-761687df3ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650783    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650843    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650908    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33fd3340-3104-450d-a434-0e89d73d7331-agent-certs\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650939    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650960    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.650975    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.651042 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651023    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651056    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651073    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651130    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651161    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651187    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651210    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651228    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651244    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651274    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651296    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651341    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb79\" (UniqueName: \"kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651431    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651456    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33fd3340-3104-450d-a434-0e89d73d7331-konnectivity-ca\") pod \"konnectivity-agent-qj2ct\" (UID: \"33fd3340-3104-450d-a434-0e89d73d7331\") " pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:48:19.651612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651526    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshvv\" (UniqueName: \"kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651644    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651703    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651729    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651749    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651764    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651780    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651835    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651870    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651908    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651949    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.651987    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652009    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69x2\" (UniqueName: \"kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652024    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652040    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652056    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.652096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652098    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652234    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652248    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652255    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652271    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652311    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652351    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652379    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652401    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c813be-5ca7-4418-891f-04939346e716-tmp-dir\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652462    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652480    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652496    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652513    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652545    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-env-overrides\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652598    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652631    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652671    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652697    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652715    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.652732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652730    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652780    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652828    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652871    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652904    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652923    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652940    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652964    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652990    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.652991    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653021    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653055    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653100    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d8677f-d924-45ce-8ff2-75fe9725346c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653114    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz4b\" (UniqueName: \"kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653172    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653236    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653272    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-textfile\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653326    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653387    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653418    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653423    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.653522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653478    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653573    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-original-pull-secret\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653944    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653985    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.653999    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.654129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654093    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.654382 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654146    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.654382 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654177    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.654382 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654210    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.654482 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654466    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovn-node-metrics-cert\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.654513 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.654481    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-ovnkube-script-lib\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.682271 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.682221    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.683393 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.683203    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:48:19.683393 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.683225    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:48:19.683528 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.683461    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:48:19.684209 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.683831    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:48:19.684209 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.683907    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:48:19.685004 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.684381    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.685004 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.684576    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:48:19.685413 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685396    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.685474 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685466    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:48:19.685524 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685489    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:48:19.685582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685406    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.685670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685652    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:48:19.686987 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685590    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:48:19.686987 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685620    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:48:19.686987 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.685583    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:48:19.689291 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689273    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.689361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689290    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.689361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689290    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.689361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689315    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.689361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689345    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-metrics-client-ca\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.689521 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689507    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.689693 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.689678    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.690713 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.690695    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.690713 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.690698    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.691813 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.691796    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.691880 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.691803    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.691880 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.691839    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.691940 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.691916    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.693030 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.693011    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/56c813be-5ca7-4418-891f-04939346e716-config-volume\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.693030 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.693021    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.694011 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.693993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.730727 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730699    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:48:19.730727 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730705    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:48:19.730881 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730763    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:48:19.730881 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730808    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:48:19.730881 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730864    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730900    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730810    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730866    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730927    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730930    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.730810    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:48:19.731057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731049    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:48:19.731388 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731181    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.731388 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731254    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:48:19.731388 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731355    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r6x\" (UniqueName: \"kubernetes.io/projected/56c813be-5ca7-4418-891f-04939346e716-kube-api-access-28r6x\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.731497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731390    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.731497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731428    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvpg\" (UniqueName: \"kubernetes.io/projected/13c429b7-91ff-41b2-96fc-575e8cc12640-kube-api-access-flvpg\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.731497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731475    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4nr\" (UniqueName: \"kubernetes.io/projected/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-kube-api-access-sz4nr\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.731599 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731536    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbvh\" (UniqueName: \"kubernetes.io/projected/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-kube-api-access-npbvh\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.731894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.731877    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:48:19.733213 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.733199    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eda94b8-a813-448e-8d29-742cd73e5d80-cert\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:48:19.733267 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.733224    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.733316 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.733297    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.733396 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.733382    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.734579 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.734560    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.737407 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.737390    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.738484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738466    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.738566 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738517    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:48:19.738566 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738558    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d7jn9\""
Nov 03 20:48:19.738680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738544    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:48:19.738680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738609    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:48:19.738680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738618    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:48:19.738680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738646    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:48:19.738680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738656    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swkc2\""
Nov 03 20:48:19.738855 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738683    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:48:19.738855 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738734    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:48:19.738855 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738763    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:48:19.738855 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738813    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:48:19.738998 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738884    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ztnjk\""
Nov 03 20:48:19.738998 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.738901    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:48:19.739105 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739079    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.739159 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739144    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:48:19.739310 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739293    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-znh84\""
Nov 03 20:48:19.739403 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739328    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q7k9s5blul5n\""
Nov 03 20:48:19.739403 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739369    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kf96\" (UniqueName: \"kubernetes.io/projected/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-kube-api-access-6kf96\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.739533 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739410    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:48:19.739533 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739446    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzzc\" (UniqueName: \"kubernetes.io/projected/3fa08991-01cf-45cd-a970-0dba2a38a851-kube-api-access-vhzzc\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.739533 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739478    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739576    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739582    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6h6b\""
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739577    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739614    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739655    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739678    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:48:19.739744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739687    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6vrj\""
Nov 03 20:48:19.740049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739782    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:48:19.740049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739876    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.740049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739883    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.740049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.739975    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.740049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740009    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740157    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740161    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740166    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb79\" (UniqueName: \"kubernetes.io/projected/983d3627-42d7-4192-ba09-c44043b657ee-kube-api-access-zsb79\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740185    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5g6d\" (UniqueName: \"kubernetes.io/projected/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-kube-api-access-v5g6d\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740171    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p9g\" (UniqueName: \"kubernetes.io/projected/c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39-kube-api-access-h2p9g\") pod \"network-check-target-h4tkw\" (UID: \"c4ca13b7-07a9-4ba0-a1c0-50f0d3a18b39\") " pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740284    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.740303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740297    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.740681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740358    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.740681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740418    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82gp\" (UniqueName: \"kubernetes.io/projected/40fb7803-b57b-4822-b933-41bfa1c40f24-kube-api-access-j82gp\") pod \"network-metrics-daemon-fjkp6\" (UID: \"40fb7803-b57b-4822-b933-41bfa1c40f24\") " pod="openshift-multus/network-metrics-daemon-fjkp6"
Nov 03 20:48:19.740768 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.740716    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.741449 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.741432    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"console-7d4747674d-vffft\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") " pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:19.741494 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.741454    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37651c6c-a9ce-4c30-af0f-7e74869a7ef4-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qxlz9\" (UID: \"37651c6c-a9ce-4c30-af0f-7e74869a7ef4\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qxlz9"
Nov 03 20:48:19.741628 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.741614    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.742370 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.742357    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-tls\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.742370 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.742364    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.742906 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.742891    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgjk\" (UniqueName: \"kubernetes.io/projected/c7d8677f-d924-45ce-8ff2-75fe9725346c-kube-api-access-6sgjk\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.743077 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.743057    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.743123 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.743061    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/56c813be-5ca7-4418-891f-04939346e716-metrics-tls\") pod \"dns-default-hjll2\" (UID: \"56c813be-5ca7-4418-891f-04939346e716\") " pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:19.743123 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.743085    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.744282 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.744265    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.744612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.744597    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69x2\" (UniqueName: \"kubernetes.io/projected/eb59a160-8501-4557-b55e-e4f3e87cfa61-kube-api-access-l69x2\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.744780 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.744765    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz4b\" (UniqueName: \"kubernetes.io/projected/7eda94b8-a813-448e-8d29-742cd73e5d80-kube-api-access-dzz4b\") pod \"ingress-canary-bznhk\" (UID: \"7eda94b8-a813-448e-8d29-742cd73e5d80\") " pod="openshift-ingress-canary/ingress-canary-bznhk"
Nov 03 20:48:19.749117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.749097    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:19.749573 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.749558    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"alertmanager-main-0\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:19.750451 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.750434    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs54\" (UniqueName: \"kubernetes.io/projected/6418ffe1-921f-4045-9e38-761687df3ad7-kube-api-access-mfs54\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.755601 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755584    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.755665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755608    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.755665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755625    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.755665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755651    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755693    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755706    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755719    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-kubelet-config\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755708    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-hostroot\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755726    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.755815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755799    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755826    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755765    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-sys-fs\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755834    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-sys\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755850    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755875    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755876    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755925    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysconfig\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755927    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-conf\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755928    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755967    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755973    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d8677f-d924-45ce-8ff2-75fe9725346c-crio-socket\") pod \"insights-runtime-extractor-mv9sd\" (UID: \"c7d8677f-d924-45ce-8ff2-75fe9725346c\") " pod="openshift-insights/insights-runtime-extractor-mv9sd"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755982    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.755997    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756018    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-sys\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756018    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756042    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-systemd-units\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756048    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756049    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756024    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756074    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-sysctl-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756085    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756110    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756124    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756091    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-device-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756148    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756174    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756178    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-bin\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756191    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756195    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756221    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-multus\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756224    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756251    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-kubelet\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756254    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756273    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756288    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756310    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756274    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-systemd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756313    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-run-ovn\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-registration-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756333    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6418ffe1-921f-4045-9e38-761687df3ad7-cnibin\") pod \"multus-additional-cni-plugins-vdkpg\" (UID: \"6418ffe1-921f-4045-9e38-761687df3ad7\") " pod="openshift-multus/multus-additional-cni-plugins-vdkpg"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-run\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756233    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-modprobe-d\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756353    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-var-lib-kubelet\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756375    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-system-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756421    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756445    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756460    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756476    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756487    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-k8s-cni-cncf-io\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756459    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-log-socket\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756510    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-cni-netd\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756526    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756536    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-slash\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756542    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756548    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec-dbus\") pod \"global-pull-secret-syncer-stvfh\" (UID: \"c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec\") " pod="kube-system/global-pull-secret-syncer-stvfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756558    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756574    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756561    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-root\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756591    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756607    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756613    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-kubernetes\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756621    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756622    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-etc-kubernetes\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756590    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-etc-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756659    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-node-log\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756662    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756682    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-multus-certs\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756681    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-run-netns\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756695    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756716    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb57a79e-5114-4c7f-97f5-23b45aebd6e7-host-slash\") pod \"iptables-alerter-s7hmx\" (UID: \"bb57a79e-5114-4c7f-97f5-23b45aebd6e7\") " pod="openshift-network-operator/iptables-alerter-s7hmx"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756719    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-etc-systemd\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756722    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756753    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756775    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756754    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-kubelet\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756806    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756813    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8dfb03f-f653-43e7-9a8a-4a61bbb2220a-node-exporter-wtmp\") pod \"node-exporter-6ssfh\" (UID: \"b8dfb03f-f653-43e7-9a8a-4a61bbb2220a\") " pod="openshift-monitoring/node-exporter-6ssfh"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756822    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756823    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-cni-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756835    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756844    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983d3627-42d7-4192-ba09-c44043b657ee-hosts-file\") pod \"node-resolver-cwmvz\" (UID: \"983d3627-42d7-4192-ba09-c44043b657ee\") " pod="openshift-dns/node-resolver-cwmvz"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756856    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756864    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-run-netns\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756872    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756892    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756906    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756910    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-cnibin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756920    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756893    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-var-lib-openvswitch\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756934    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756874    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-conf-dir\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756963    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-host-var-lib-cni-bin\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756978    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13c429b7-91ff-41b2-96fc-575e8cc12640-socket-dir\") pod \"aws-ebs-csi-driver-node-v2sfn\" (UID: \"13c429b7-91ff-41b2-96fc-575e8cc12640\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2sfn"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.756983    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb59a160-8501-4557-b55e-e4f3e87cfa61-host\") pod \"node-ca-mwqr2\" (UID: \"eb59a160-8501-4557-b55e-e4f3e87cfa61\") " pod="openshift-image-registry/node-ca-mwqr2"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757008    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.756976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757024    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.759014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757038    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.759014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757052    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-lib-modules\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.759014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757055    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fc55c3-89a7-4d55-bae4-a64d15efebaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n99n6\" (UID: \"c8fc55c3-89a7-4d55-bae4-a64d15efebaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:19.759014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757069    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa08991-01cf-45cd-a970-0dba2a38a851-host\") pod \"tuned-9thx7\" (UID: \"3fa08991-01cf-45cd-a970-0dba2a38a851\") " pod="openshift-cluster-node-tuning-operator/tuned-9thx7"
Nov 03 20:48:19.759014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.757092    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b1c9b27-49d4-4d22-9ff7-350c94e96665-multus-socket-dir-parent\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.759429 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.759396    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshvv\" (UniqueName: \"kubernetes.io/projected/8b1c9b27-49d4-4d22-9ff7-350c94e96665-kube-api-access-cshvv\") pod \"multus-8kvc5\" (UID: \"8b1c9b27-49d4-4d22-9ff7-350c94e96665\") " pod="openshift-multus/multus-8kvc5"
Nov 03 20:48:19.766146 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.766130    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"console-86f484b7d7-ptsf7\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") " pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:19.842320 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.842304    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:48:19.904818 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:19.904793    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:20.600678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:20.600647    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:48:21.294653 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:21.294600    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hjll2"
Nov 03 20:48:21.541747 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:21.541718    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:22.224061 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.224036    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:22.242308 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.242287    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:22.564748 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.564680    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerStarted","Data":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"}
Nov 03 20:48:22.564748 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.564742    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerStarted","Data":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"}
Nov 03 20:48:22.564898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.564769    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerStarted","Data":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"}
Nov 03 20:48:22.564898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:22.564792    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerStarted","Data":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"}
Nov 03 20:48:24.617486 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:24.617450    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:25.385871 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:25.385830    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:26.555111 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:26.555084    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:26.555482 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:26.555304    5533 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:48:27.805177 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:27.805141    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:27.926972 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:27.926944    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n99n6"
Nov 03 20:48:29.909448 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:29.909367    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:29.913258 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:29.913240    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:48:43.935715 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:43.935684    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qj2ct"
Nov 03 20:48:45.620418 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.620358    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86f484b7d7-ptsf7" podUID="f3b31527-43de-413d-8b90-962283a47555" containerName="console" containerID="cri-o://2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f" gracePeriod=15
Nov 03 20:48:45.888155 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.888107    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:45.977619 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977593    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977741 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977678    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977741 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977717    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977821 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977739    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977821 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977766    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977913 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977822    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.977913 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.977851    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") pod \"f3b31527-43de-413d-8b90-962283a47555\" (UID: \"f3b31527-43de-413d-8b90-962283a47555\") "
Nov 03 20:48:45.978146 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.978075    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:45.978330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.978177    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:45.978330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.978284    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:45.978330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.978312    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:45.979988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.979953    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:45.979988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.979976    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "kube-api-access-tzp26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:45.980114 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:45.980004    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config" pod "f3b31527-43de-413d-8b90-962283a47555" (UID: "f3b31527-43de-413d-8b90-962283a47555"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:46.079145 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079117    5533 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b31527-43de-413d-8b90-962283a47555-tmp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079145 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079142    5533 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-console-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079145 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079152    5533 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079162    5533 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-serving-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079171    5533 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3b31527-43de-413d-8b90-962283a47555-console-oauth-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079181    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp26\" (UniqueName: \"kubernetes.io/projected/f3b31527-43de-413d-8b90-962283a47555-kube-api-access-tzp26\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.079191    5533 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3b31527-43de-413d-8b90-962283a47555-oauth-serving-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:46.645325 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.645299    5533 generic.go:356] "Generic (PLEG): container finished" podID="f3b31527-43de-413d-8b90-962283a47555" containerID="2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f" exitCode=0
Nov 03 20:48:46.645815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.645353    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f484b7d7-ptsf7"
Nov 03 20:48:46.645815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.645383    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f484b7d7-ptsf7" event={"ID":"f3b31527-43de-413d-8b90-962283a47555","Type":"ContainerDied","Data":"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"}
Nov 03 20:48:46.645815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.645434    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f484b7d7-ptsf7" event={"ID":"f3b31527-43de-413d-8b90-962283a47555","Type":"ContainerDied","Data":"2fe9a01eb14b11ffb734ac5b52ccffa18fd7a2a0ff71081df884e0830d40ab8a"}
Nov 03 20:48:46.645815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.645464    5533 scope.go:122] "RemoveContainer" containerID="2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"
Nov 03 20:48:46.654896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.653845    5533 scope.go:122] "RemoveContainer" containerID="2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"
Nov 03 20:48:46.655282 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:46.655248    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f\": container with ID starting with 2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f not found: ID does not exist" containerID="2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"
Nov 03 20:48:46.655345 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.655290    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f"} err="failed to get container status \"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f\": rpc error: code = NotFound desc = could not find container \"2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f\": container with ID starting with 2c2748a6b9d1ec2e48a208905583db3800defda9b0a6702e8438431cbddf092f not found: ID does not exist"
Nov 03 20:48:46.798920 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.798877    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:48:46.845715 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:46.845693    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f484b7d7-ptsf7"]
Nov 03 20:48:48.490179 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:48.490148    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b31527-43de-413d-8b90-962283a47555" path="/var/lib/kubelet/pods/f3b31527-43de-413d-8b90-962283a47555/volumes"
Nov 03 20:48:49.049205 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:49.049176    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:48:49.258941 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:48:49.258910    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:48:55.389463 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:55.389425    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4tkw"
Nov 03 20:48:58.510651 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.510599    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:48:58.511044 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.510925    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="alertmanager" containerID="cri-o://16235897dbfb73b5f4fd59b1d80b3719a0e0703f55958fe480d116a6415b7eec" gracePeriod=120
Nov 03 20:48:58.511044 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.510974    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-metric" containerID="cri-o://b783b37f381dfc475c8846752cc3994877808b8cd8fcc4fb9e697be63d84427b" gracePeriod=120
Nov 03 20:48:58.511191 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.511016    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-web" containerID="cri-o://b1608b39de685bffc58433b152c25b12d3130da9a8fb78785f2ce7e71b2b9e68" gracePeriod=120
Nov 03 20:48:58.511191 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.511036    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="prom-label-proxy" containerID="cri-o://7ffd7bce383178d7c6e69a0ae84c9c8379be065e56345a4158ecef016098883b" gracePeriod=120
Nov 03 20:48:58.511191 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.511120    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="config-reloader" containerID="cri-o://1e6aa412339e22b13b9a9c172abf8c7f731d8a0b51fde823bb4de9bfb1bd44d2" gracePeriod=120
Nov 03 20:48:58.511191 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.511052    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy" containerID="cri-o://d615301ad3171733421c15b3123d4aa3f2920ba3b4f70529159be406a9baaf66" gracePeriod=120
Nov 03 20:48:58.707143 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707119    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="7ffd7bce383178d7c6e69a0ae84c9c8379be065e56345a4158ecef016098883b" exitCode=0
Nov 03 20:48:58.707143 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707140    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="b783b37f381dfc475c8846752cc3994877808b8cd8fcc4fb9e697be63d84427b" exitCode=0
Nov 03 20:48:58.707143 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707146    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="d615301ad3171733421c15b3123d4aa3f2920ba3b4f70529159be406a9baaf66" exitCode=0
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707152    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="1e6aa412339e22b13b9a9c172abf8c7f731d8a0b51fde823bb4de9bfb1bd44d2" exitCode=0
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707158    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="16235897dbfb73b5f4fd59b1d80b3719a0e0703f55958fe480d116a6415b7eec" exitCode=0
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707185    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"7ffd7bce383178d7c6e69a0ae84c9c8379be065e56345a4158ecef016098883b"}
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707240    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"b783b37f381dfc475c8846752cc3994877808b8cd8fcc4fb9e697be63d84427b"}
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707265    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"d615301ad3171733421c15b3123d4aa3f2920ba3b4f70529159be406a9baaf66"}
Nov 03 20:48:58.707300 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707289    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"1e6aa412339e22b13b9a9c172abf8c7f731d8a0b51fde823bb4de9bfb1bd44d2"}
Nov 03 20:48:58.707470 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:58.707312    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"16235897dbfb73b5f4fd59b1d80b3719a0e0703f55958fe480d116a6415b7eec"}
Nov 03 20:48:59.715004 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.714976    5533 generic.go:356] "Generic (PLEG): container finished" podID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerID="b1608b39de685bffc58433b152c25b12d3130da9a8fb78785f2ce7e71b2b9e68" exitCode=0
Nov 03 20:48:59.715333 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.715027    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"b1608b39de685bffc58433b152c25b12d3130da9a8fb78785f2ce7e71b2b9e68"}
Nov 03 20:48:59.762378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.762357    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:48:59.888665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888592    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888625    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888664    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888683    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888725    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888757    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888779    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888832    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.888894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888861    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.889192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888900    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.889192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888934    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.889192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.888984    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.889192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.889019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") pod \"1195581f-22f9-447f-8f94-c3a08eaa48a1\" (UID: \"1195581f-22f9-447f-8f94-c3a08eaa48a1\") "
Nov 03 20:48:59.889192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.889101    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:59.889483 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.889323    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:59.889483 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.889335    5533 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-main-db\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.889976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.889942    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:59.892230 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.892202    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.892527 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.892497    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.892670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.892626    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.892670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.892662    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:59.892868 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.892845    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:59.893376 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.893350    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "kube-api-access-rw5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:59.893588 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.893563    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.893726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.893681    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.896404 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.896384    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.902097 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.902072    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config" pod "1195581f-22f9-447f-8f94-c3a08eaa48a1" (UID: "1195581f-22f9-447f-8f94-c3a08eaa48a1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:59.990215 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990177    5533 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-web-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990220    5533 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-main-tls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990239    5533 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990254    5533 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-tls-assets\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990268    5533 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-cluster-tls-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990283    5533 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1195581f-22f9-447f-8f94-c3a08eaa48a1-config-out\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990298    5533 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990313    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5wf\" (UniqueName: \"kubernetes.io/projected/1195581f-22f9-447f-8f94-c3a08eaa48a1-kube-api-access-rw5wf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990327    5533 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-metrics-client-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990342    5533 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1195581f-22f9-447f-8f94-c3a08eaa48a1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990364 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990358    5533 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:59.990666 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:48:59.990372    5533 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1195581f-22f9-447f-8f94-c3a08eaa48a1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:00.720678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.720628    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1195581f-22f9-447f-8f94-c3a08eaa48a1","Type":"ContainerDied","Data":"c72a604e6f2d27668fe64924e57bc631d05919ac4d37894d15380f9adf7471f9"}
Nov 03 20:49:00.721089 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.720701    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:49:00.721089 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.720718    5533 scope.go:122] "RemoveContainer" containerID="7ffd7bce383178d7c6e69a0ae84c9c8379be065e56345a4158ecef016098883b"
Nov 03 20:49:00.727906 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.727889    5533 scope.go:122] "RemoveContainer" containerID="b783b37f381dfc475c8846752cc3994877808b8cd8fcc4fb9e697be63d84427b"
Nov 03 20:49:00.734510 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.734491    5533 scope.go:122] "RemoveContainer" containerID="d615301ad3171733421c15b3123d4aa3f2920ba3b4f70529159be406a9baaf66"
Nov 03 20:49:00.740657 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.740621    5533 scope.go:122] "RemoveContainer" containerID="b1608b39de685bffc58433b152c25b12d3130da9a8fb78785f2ce7e71b2b9e68"
Nov 03 20:49:00.746854 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.746835    5533 scope.go:122] "RemoveContainer" containerID="1e6aa412339e22b13b9a9c172abf8c7f731d8a0b51fde823bb4de9bfb1bd44d2"
Nov 03 20:49:00.753327 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.753304    5533 scope.go:122] "RemoveContainer" containerID="16235897dbfb73b5f4fd59b1d80b3719a0e0703f55958fe480d116a6415b7eec"
Nov 03 20:49:00.759701 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.759685    5533 scope.go:122] "RemoveContainer" containerID="85420996b2d1fef7e51fd505237eda959587a19601509d4abae8f6f595307878"
Nov 03 20:49:00.924559 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.924533    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:49:00.971469 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:00.971436    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:49:02.490403 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:02.490368    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" path="/var/lib/kubelet/pods/1195581f-22f9-447f-8f94-c3a08eaa48a1/volumes"
Nov 03 20:49:03.996354 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:03.996322    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:49:04.274054 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.273982    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:04.274306 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274278    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="prometheus" containerID="cri-o://28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74" gracePeriod=600
Nov 03 20:49:04.274384 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274313    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="thanos-sidecar" containerID="cri-o://a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106" gracePeriod=600
Nov 03 20:49:04.274384 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274327    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy" containerID="cri-o://21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d" gracePeriod=600
Nov 03 20:49:04.274497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274362    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="config-reloader" containerID="cri-o://62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef" gracePeriod=600
Nov 03 20:49:04.274497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274427    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-web" containerID="cri-o://27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726" gracePeriod=600
Nov 03 20:49:04.274497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.274358    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428" gracePeriod=600
Nov 03 20:49:04.517403 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.517379    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:04.622683 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622597    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622683 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622629    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622683 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622677    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622927 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622702    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622927 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622728    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622927 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622768    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.622927 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622853    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623082 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622946    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623082 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.622980    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623082 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623015    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623082 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623057    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623097    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623125    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623151    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623177    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623195    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623200    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623221    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623258    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623279    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623304    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") pod \"943e14bc-5d36-4152-8049-585724259f3f\" (UID: \"943e14bc-5d36-4152-8049-585724259f3f\") "
Nov 03 20:49:04.623680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623460    5533 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.624331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.623860    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.624462 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.624389    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.624524    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.624854    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.625371    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.625488    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.625607    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.625806    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.625985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.625959    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.627238 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.627211    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.627337 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.627232    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "kube-api-access-z2xfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:04.627862 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.627816    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.627862 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.627819    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:04.628035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.627874    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.628098 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.628070    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.628372 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.628353    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.628582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.628563    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.628848 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.628825    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:04.635785 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.635760    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config" pod "943e14bc-5d36-4152-8049-585724259f3f" (UID: "943e14bc-5d36-4152-8049-585724259f3f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:04.724247 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724221    5533 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724247 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724245    5533 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724256    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xfh\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-kube-api-access-z2xfh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724265    5533 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724274    5533 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-metrics-client-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724283    5533 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724292    5533 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-web-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724301    5533 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-metrics-client-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724311    5533 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724319    5533 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724328    5533 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-kube-rbac-proxy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724337    5533 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-db\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724345    5533 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724355    5533 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/943e14bc-5d36-4152-8049-585724259f3f-config-out\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724363    5533 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/943e14bc-5d36-4152-8049-585724259f3f-tls-assets\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724371    5533 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724381    5533 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-secret-grpc-tls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724808 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724390    5533 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/943e14bc-5d36-4152-8049-585724259f3f-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.724808 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.724398    5533 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/943e14bc-5d36-4152-8049-585724259f3f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:04.738104 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738078    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428" exitCode=0
Nov 03 20:49:04.738104 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738100    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d" exitCode=0
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738108    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726" exitCode=0
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738117    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106" exitCode=0
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738126    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef" exitCode=0
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738134    5533 generic.go:356] "Generic (PLEG): container finished" podID="943e14bc-5d36-4152-8049-585724259f3f" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74" exitCode=0
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738135    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"}
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738166    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738200    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"}
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738227    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"}
Nov 03 20:49:04.738260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738250    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"}
Nov 03 20:49:04.738726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738268    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.738726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738274    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"}
Nov 03 20:49:04.738726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738298    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"}
Nov 03 20:49:04.738726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.738321    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"943e14bc-5d36-4152-8049-585724259f3f","Type":"ContainerDied","Data":"7f801c0cff104a47f0558712e901749217648e53bdc3277b0fd21361bd76b926"}
Nov 03 20:49:04.745448 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.745428    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.751988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.751970    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.757919 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.757903    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.763773 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.763753    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.769625 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.769608    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.775978 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.775962    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.781569 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.781555    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.781829 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.781810    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.781879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.781836    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.781879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.781853    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.782055 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.782040    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.782093 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782060    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.782093 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782073    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.782290 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.782274    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.782330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782294    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.782330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782306    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.782488 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.782474    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.782527 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782491    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.782527 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782502    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.782769 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.782752    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.782824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782772    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.782824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782793    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.782993 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.782978    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.783028 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.782996    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.783028 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783007    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.783203 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:04.783189    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.783239 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783207    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.783239 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783219    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.783420 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783404    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.783460 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783419    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.783604 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783589    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.783661 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783604    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.783799 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783784    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.783799 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783798    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.783972 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783956    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.783972 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.783971    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.784145 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784130    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.784185 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784144    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.784308 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784293    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.784354 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784310    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.784471 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784456    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.784511 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784470    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.784658 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784620    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.784704 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784658    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.784824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784809    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.784862 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784824    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.784985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784970    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.784985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.784984    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.785149 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785133    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.785149 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785148    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.785333 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785318    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.785373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785333    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.785510 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785495    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.785546 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785510    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.785707 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785690    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.785707 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785706    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.785921 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785900    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.785971 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.785922    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.786168 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786152    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.786168 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786167    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.786402 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786385    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.786402 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786402    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.786612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786597    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.786612 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786612    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.786837 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786821    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.786837 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.786836    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.787040 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787025    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.787092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787040    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.787233 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787218    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.787274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787232    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.787404 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787384    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.787445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787405    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.787613 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787598    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.787732 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787612    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.787872 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787856    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.787906 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.787872    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.788079 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788055    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.788142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788080    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.788281 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788263    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.788281 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788280    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.788472 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788449    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.788517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788471    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.788758 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788740    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.788835 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.788759    5533 scope.go:122] "RemoveContainer" containerID="93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"
Nov 03 20:49:04.789022 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789002    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428"} err="failed to get container status \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": rpc error: code = NotFound desc = could not find container \"93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428\": container with ID starting with 93c53df303b6c8b33101990edf6bbea8318ff06324080197679f27a4662ef428 not found: ID does not exist"
Nov 03 20:49:04.789089 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789024    5533 scope.go:122] "RemoveContainer" containerID="21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"
Nov 03 20:49:04.789256 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789231    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d"} err="failed to get container status \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": rpc error: code = NotFound desc = could not find container \"21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d\": container with ID starting with 21ca0db67802ea1b0e87f07f9377a585f96814dd597ed04adbcc9a57913e8d8d not found: ID does not exist"
Nov 03 20:49:04.789306 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789256    5533 scope.go:122] "RemoveContainer" containerID="27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"
Nov 03 20:49:04.789498 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789480    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726"} err="failed to get container status \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": rpc error: code = NotFound desc = could not find container \"27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726\": container with ID starting with 27b89fa84ffd5a86e0035f86d8a4e25c154c52ecc21e5eaee6e69dbcfa959726 not found: ID does not exist"
Nov 03 20:49:04.789498 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789498    5533 scope.go:122] "RemoveContainer" containerID="a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"
Nov 03 20:49:04.789740 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789722    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106"} err="failed to get container status \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": rpc error: code = NotFound desc = could not find container \"a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106\": container with ID starting with a504829983ae264996d2a3bab43276b94e6c3923f98a14daee4718ae8f804106 not found: ID does not exist"
Nov 03 20:49:04.789789 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789741    5533 scope.go:122] "RemoveContainer" containerID="62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"
Nov 03 20:49:04.789945 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789926    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef"} err="failed to get container status \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": rpc error: code = NotFound desc = could not find container \"62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef\": container with ID starting with 62e997a481dd0f7fdefa5978d4c670d47523d674a2c7791682446eff8e5dd6ef not found: ID does not exist"
Nov 03 20:49:04.789985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.789945    5533 scope.go:122] "RemoveContainer" containerID="28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"
Nov 03 20:49:04.790172 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.790146    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74"} err="failed to get container status \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": rpc error: code = NotFound desc = could not find container \"28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74\": container with ID starting with 28b1afd90933e40f115b3fdfd2b435f1202a77cde738cddffca083ff00201c74 not found: ID does not exist"
Nov 03 20:49:04.790215 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.790172    5533 scope.go:122] "RemoveContainer" containerID="96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"
Nov 03 20:49:04.790374 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.790356    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da"} err="failed to get container status \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": rpc error: code = NotFound desc = could not find container \"96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da\": container with ID starting with 96b6060dd860c646c0e7c5d9966d4d5403a5540e77a7dfd42e362f55065cf1da not found: ID does not exist"
Nov 03 20:49:04.896022 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.895974    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:04.899971 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.899943    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934323    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934598    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f3b31527-43de-413d-8b90-962283a47555" containerName="console"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934611    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b31527-43de-413d-8b90-962283a47555" containerName="console"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934628    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="init-config-reloader"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934659    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="init-config-reloader"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934670    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-metric"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934680    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-metric"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934691    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="thanos-sidecar"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934699    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="thanos-sidecar"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934712    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934719    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934732    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934739    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934749    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-thanos"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934756    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-thanos"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934768    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="prom-label-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934775    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="prom-label-proxy"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934790    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="init-config-reloader"
Nov 03 20:49:04.934845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934798    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="init-config-reloader"
Nov 03 20:49:04.935879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.934812    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="prometheus"
Nov 03 20:49:04.935879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.935822    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="prometheus"
Nov 03 20:49:04.935879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.935837    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.935879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.935845    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.935879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.935855    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="config-reloader"
Nov 03 20:49:04.936208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.935862    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="config-reloader"
Nov 03 20:49:04.936208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936147    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="alertmanager"
Nov 03 20:49:04.936208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936157    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="alertmanager"
Nov 03 20:49:04.936208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936175    5533 cpu_manager.go:414] "containerMap: removing container" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.936208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936183    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.936537 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936193    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="config-reloader"
Nov 03 20:49:04.936537 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936473    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="config-reloader"
Nov 03 20:49:04.936740 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936706    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="init-config-reloader"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936723    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-metric"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936829    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="thanos-sidecar"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936839    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936849    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936857    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-thanos"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936867    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="prom-label-proxy"
Nov 03 20:49:04.936896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.936877    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="prometheus"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937235    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937251    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="config-reloader"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937259    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="alertmanager"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937268    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="943e14bc-5d36-4152-8049-585724259f3f" containerName="kube-rbac-proxy-web"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937277    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195581f-22f9-447f-8f94-c3a08eaa48a1" containerName="config-reloader"
Nov 03 20:49:04.937309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.937284    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b31527-43de-413d-8b90-962283a47555" containerName="console"
Nov 03 20:49:04.942618 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.942558    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:04.992383 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992342    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:49:04.992383 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992372    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:49:04.992565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992386    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:49:04.992565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992416    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:49:04.992565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992454    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:49:04.992565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992459    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m6vrj\""
Nov 03 20:49:04.992565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.992509    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5q7k9s5blul5n\""
Nov 03 20:49:04.993207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993191    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:49:04.993390 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993377    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:49:04.993480 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993409    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:49:04.993517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993497    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:49:04.993568 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993537    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:49:04.993568 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.993541    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:49:04.994580 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.994568    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:49:04.994645 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:04.994602    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:49:05.040378 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.040355    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:49:05.040673 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.040496    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:49:05.041278 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.041259    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:05.127605 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127579    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127714 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127615    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127714 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127664    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127714 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127702    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127834 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127881 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127842    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127881 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127865    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127944    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.127988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127975    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128080 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.127999    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128116 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128089    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128116 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128113    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128187 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128134    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128187 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128153    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct6k\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-kube-api-access-xct6k\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128184    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128260 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128245    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128326 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128264    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128326 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128282    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128326 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128297    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.128416 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.128325    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229308 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229278    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229402 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229323    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229402 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229355    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229402 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229384    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229408    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229431    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229466    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229518    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229542    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229567    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229609    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229658    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229689    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229713    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229736    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229769    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.229869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229777    5533 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:49:05.230313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.230060    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.230313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.230079    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.230414 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.230315    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.230414 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.229801    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.230414 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.230391    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.230554 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.230423    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct6k\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-kube-api-access-xct6k\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.231465 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.231437    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.231690 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.231668    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.232004 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.231976    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.232087 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.232022    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.234073 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.234047    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/225bfb37-e79e-4293-ad0f-befcfc42f1ae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.235351 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.235304    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.235505 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.235486    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236591 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236296    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config-out\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236591 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236427    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236591 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236543    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236809 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236686    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236809 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236795    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.236929 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.236794    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.237071 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.237049    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.237719 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.237698    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.237940 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.237924    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/225bfb37-e79e-4293-ad0f-befcfc42f1ae-web-config\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.290129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.290094    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct6k\" (UniqueName: \"kubernetes.io/projected/225bfb37-e79e-4293-ad0f-befcfc42f1ae-kube-api-access-xct6k\") pod \"prometheus-k8s-0\" (UID: \"225bfb37-e79e-4293-ad0f-befcfc42f1ae\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:05.567006 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:05.566931    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:06.034397 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:06.034369    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:49:06.037698 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:49:06.037616    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225bfb37_e79e_4293_ad0f_befcfc42f1ae.slice/crio-cea289b8587cb8f1a3cb38b8fdf96b70fc28fab3de2acde8d83b9ce399a3aa92 WatchSource:0}: Error finding container cea289b8587cb8f1a3cb38b8fdf96b70fc28fab3de2acde8d83b9ce399a3aa92: Status 404 returned error can't find the container with id cea289b8587cb8f1a3cb38b8fdf96b70fc28fab3de2acde8d83b9ce399a3aa92
Nov 03 20:49:06.490766 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:06.490734    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943e14bc-5d36-4152-8049-585724259f3f" path="/var/lib/kubelet/pods/943e14bc-5d36-4152-8049-585724259f3f/volumes"
Nov 03 20:49:06.746879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:06.746821    5533 generic.go:356] "Generic (PLEG): container finished" podID="225bfb37-e79e-4293-ad0f-befcfc42f1ae" containerID="2aafe2bf752e18cbc4a7b6e0d8a25fa6c4239f3ff0797ce2f331d42641d2a229" exitCode=0
Nov 03 20:49:06.746998 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:06.746912    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerDied","Data":"2aafe2bf752e18cbc4a7b6e0d8a25fa6c4239f3ff0797ce2f331d42641d2a229"}
Nov 03 20:49:06.747038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:06.747000    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"cea289b8587cb8f1a3cb38b8fdf96b70fc28fab3de2acde8d83b9ce399a3aa92"}
Nov 03 20:49:07.755549 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755516    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"c649535f97c7b1acb14ac64ed3d04a4f08f2eafe1897872625ab4d337ef71cb5"}
Nov 03 20:49:07.755954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755607    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"5aafd401f3a522775a28c5fe4df1cbdb843df6920a50d32a2f81eaf44f385c05"}
Nov 03 20:49:07.755954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755654    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"929e6d71e276ce922f18959cbf080a9ca6d2c420d44712136c8333150cd7e9eb"}
Nov 03 20:49:07.755954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755679    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"db5eb17aee1853812841e233016baf972a2319e5d693fb5eba1e91386b042b4a"}
Nov 03 20:49:07.755954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755704    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"3dfc5032f48223188856b964518b1c616b8da1192f3f56c9a5175ed9df7d7948"}
Nov 03 20:49:07.755954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.755727    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"225bfb37-e79e-4293-ad0f-befcfc42f1ae","Type":"ContainerStarted","Data":"d146db32d78c9e182da70aaa853baf3a6fb8548b951665d9e3a5280b805fe040"}
Nov 03 20:49:07.881954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:07.881899    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.8648343670000003 podStartE2EDuration="3.881885783s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.051416ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:07.881233832 +0000 UTC m=+49.512547341" watchObservedRunningTime="2026-11-03 20:49:07.881885783 +0000 UTC m=+49.513199254"
Nov 03 20:49:10.567221 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:10.567184    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:16.929314 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:16.929269    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:16.944852 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:16.944827    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:17.806478 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:17.806448    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:49:19.261959 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:19.261919    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:49:29.014946 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.014909    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d4747674d-vffft" podUID="714afbba-883e-46b4-927f-473d9d940307" containerName="console" containerID="cri-o://65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958" gracePeriod=15
Nov 03 20:49:29.255133 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.255110    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:49:29.296704 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296651    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296704 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296684    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296705    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296733    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296751    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296791    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296983 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296897    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.296983 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.296966    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") pod \"714afbba-883e-46b4-927f-473d9d940307\" (UID: \"714afbba-883e-46b4-927f-473d9d940307\") "
Nov 03 20:49:29.297361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297079    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:29.297361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297098    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:29.297361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297167    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:29.297361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297171    5533 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/714afbba-883e-46b4-927f-473d9d940307-tmp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.297361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297273    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:29.297564 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.297379    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:29.298884 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.298853    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:29.299048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.299028    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:29.299048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.299033    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk" pod "714afbba-883e-46b4-927f-473d9d940307" (UID: "714afbba-883e-46b4-927f-473d9d940307"). InnerVolumeSpecName "kube-api-access-kddtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:29.397841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397817    5533 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-oauth-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397841 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397840    5533 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397994 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397850    5533 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-trusted-ca-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397994 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397858    5533 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/714afbba-883e-46b4-927f-473d9d940307-console-serving-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397994 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397868    5533 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-oauth-serving-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397994 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397878    5533 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/714afbba-883e-46b4-927f-473d9d940307-console-config\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.397994 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.397888    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddtk\" (UniqueName: \"kubernetes.io/projected/714afbba-883e-46b4-927f-473d9d940307-kube-api-access-kddtk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:29.829510 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.829474    5533 generic.go:356] "Generic (PLEG): container finished" podID="714afbba-883e-46b4-927f-473d9d940307" containerID="65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958" exitCode=0
Nov 03 20:49:29.829693 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.829519    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4747674d-vffft" event={"ID":"714afbba-883e-46b4-927f-473d9d940307","Type":"ContainerDied","Data":"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"}
Nov 03 20:49:29.829693 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.829527    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4747674d-vffft"
Nov 03 20:49:29.829693 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.829561    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4747674d-vffft" event={"ID":"714afbba-883e-46b4-927f-473d9d940307","Type":"ContainerDied","Data":"14e9e58bf0b23243ab0d64161340fb7c1ad705e407b43175a0445043f6517f81"}
Nov 03 20:49:29.829693 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.829579    5533 scope.go:122] "RemoveContainer" containerID="65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"
Nov 03 20:49:29.837276 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.837257    5533 scope.go:122] "RemoveContainer" containerID="65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"
Nov 03 20:49:29.837536 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:29.837516    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958\": container with ID starting with 65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958 not found: ID does not exist" containerID="65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"
Nov 03 20:49:29.837580 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.837544    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958"} err="failed to get container status \"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958\": rpc error: code = NotFound desc = could not find container \"65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958\": container with ID starting with 65eb44f29e5b0280b3caa3a0041a38cbaa6a66200a3887bc0ce7cbfc1eb98958 not found: ID does not exist"
Nov 03 20:49:29.989169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:29.989136    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:49:30.036112 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:30.036085    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4747674d-vffft"]
Nov 03 20:49:30.490278 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:49:30.490235    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714afbba-883e-46b4-927f-473d9d940307" path="/var/lib/kubelet/pods/714afbba-883e-46b4-927f-473d9d940307/volumes"
Nov 03 20:49:49.258957 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:49:49.258917    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:50:19.261064 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:50:19.261028    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:50:49.265274 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:50:49.265226    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:51:19.260344 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:51:19.260302    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:51:49.259927 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:51:49.259891    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:52:18.111432 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.111398    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"]
Nov 03 20:52:18.113816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.111667    5533 cpu_manager.go:414] "containerMap: removing container" podUID="714afbba-883e-46b4-927f-473d9d940307" containerName="console"
Nov 03 20:52:18.113816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.111679    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714afbba-883e-46b4-927f-473d9d940307" containerName="console"
Nov 03 20:52:18.113816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.111744    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="714afbba-883e-46b4-927f-473d9d940307" containerName="console"
Nov 03 20:52:18.114650 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.114612    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.210313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.210286    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxdc\" (UniqueName: \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.210419 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.210331    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.210419 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.210412    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.211470 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.211446    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-v424j\""
Nov 03 20:52:18.211520 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.211485    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:52:18.211681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.211661    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:52:18.212247 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.212230    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"]
Nov 03 20:52:18.212567 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.212550    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"]
Nov 03 20:52:18.215698 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.215680    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.265039 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.265020    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pmxkh\""
Nov 03 20:52:18.276406 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.276303    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"]
Nov 03 20:52:18.307581 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.307549    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"]
Nov 03 20:52:18.310472 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310457    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.310723 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310702    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.310815 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4f7\" (UniqueName: \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.310866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310846    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.310910 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310898    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.310953 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310935    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxdc\" (UniqueName: \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.311000 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.310977    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.311187 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.311167    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.311252 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.311233    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.366593 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.366525    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h9rzj\""
Nov 03 20:52:18.367675 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.367625    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"]
Nov 03 20:52:18.373853 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.373833    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxdc\" (UniqueName: \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.410770 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.410743    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"]
Nov 03 20:52:18.411421 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411402    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.411479 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411445    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.411523 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411479    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.411559 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411531    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.411751 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411730    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv888\" (UniqueName: \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.411825 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411784    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4f7\" (UniqueName: \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.411825 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411791    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.411927 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.411854    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.414117 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.414096    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.429274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.429256    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:18.465285 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.465262    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-m6qvc\""
Nov 03 20:52:18.469843 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.469818    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"]
Nov 03 20:52:18.472823 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.472797    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4f7\" (UniqueName: \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.512530 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512506    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.512660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512543    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.512660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512572    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwprv\" (UniqueName: \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.512660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512618    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.512660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512655    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.512849 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512828    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv888\" (UniqueName: \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.512940 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512920    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.512974 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.512953    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.572313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.572287    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv888\" (UniqueName: \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.578354 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.578336    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pmxkh\""
Nov 03 20:52:18.580468 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.580452    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:18.613773 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.613740    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.613894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.613776    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwprv\" (UniqueName: \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.613894 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.613802    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.614086 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.614068    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.614154 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.614096    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.671246 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.671219    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwprv\" (UniqueName: \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.673889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.673872    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h9rzj\""
Nov 03 20:52:18.675914 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.675899    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:18.683236 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.683206    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"]
Nov 03 20:52:18.686667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.686649    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.730919 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.730874    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:18.736208 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.736183    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-c8pfz\""
Nov 03 20:52:18.744151 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.742270    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"]
Nov 03 20:52:18.793565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.793540    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"]
Nov 03 20:52:18.796497 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:18.796470    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9961fc23_444d_4279_b442_ca20168d2a56.slice/crio-521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23 WatchSource:0}: Error finding container 521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23: Status 404 returned error can't find the container with id 521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23
Nov 03 20:52:18.816169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.816145    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.816248 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.816182    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.816296 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.816264    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7rh\" (UniqueName: \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.916796 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.916744    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.916796 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.916783    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.916918 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.916831    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7rh\" (UniqueName: \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.917097 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.917080    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.917176 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.917161    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:18.945551 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.945530    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"]
Nov 03 20:52:18.948494 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:18.948464    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad1ea12_59b1_4471_adee_979d8f15802e.slice/crio-4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f WatchSource:0}: Error finding container 4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f: Status 404 returned error can't find the container with id 4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f
Nov 03 20:52:18.974780 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:18.974759    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7rh\" (UniqueName: \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:19.002387 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.002371    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:19.036786 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.036759    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"]
Nov 03 20:52:19.040941 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:19.040920    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404d2d4a_3b45_413a_9a44_b2ec6bddcf3f.slice/crio-170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a WatchSource:0}: Error finding container 170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a: Status 404 returned error can't find the container with id 170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a
Nov 03 20:52:19.096615 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.096591    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"]
Nov 03 20:52:19.099554 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:19.099530    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bcf0cc_19d3_4896_b41a_d5e8eee25017.slice/crio-3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f WatchSource:0}: Error finding container 3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f: Status 404 returned error can't find the container with id 3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f
Nov 03 20:52:19.259130 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:52:19.259103    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:52:19.363187 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.363163    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"]
Nov 03 20:52:19.366497 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:19.366472    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadf81b0_3b0b_4b9b_ac77_fdb73213b676.slice/crio-5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e WatchSource:0}: Error finding container 5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e: Status 404 returned error can't find the container with id 5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e
Nov 03 20:52:19.371109 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.371085    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerStarted","Data":"5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e"}
Nov 03 20:52:19.372147 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.372122    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf" event={"ID":"83bcf0cc-19d3-4896-b41a-d5e8eee25017","Type":"ContainerStarted","Data":"3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f"}
Nov 03 20:52:19.373195 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.373176    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh" event={"ID":"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f","Type":"ContainerStarted","Data":"170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a"}
Nov 03 20:52:19.374275 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.374249    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerStarted","Data":"4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f"}
Nov 03 20:52:19.375357 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:19.375338    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerStarted","Data":"521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23"}
Nov 03 20:52:20.452107 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.452065    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"]
Nov 03 20:52:20.456005 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.455983    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.506452 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.506425    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-68dwf\""
Nov 03 20:52:20.510726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.510704    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"]
Nov 03 20:52:20.532293 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.532262    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.532407 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.532333    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6tx\" (UniqueName: \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.532407 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.532367    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.632920 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.632886    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6tx\" (UniqueName: \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.633035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.632937    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.633035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.633014    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.633427 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.633403    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.633987 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.633963    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.706366 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.706310    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6tx\" (UniqueName: \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:20.776527 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:20.776502    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:21.183896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:21.181888    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"]
Nov 03 20:52:21.185982 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:21.185948    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09523415_47a1_437b_b5c5_b8a4350fed2e.slice/crio-74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485 WatchSource:0}: Error finding container 74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485: Status 404 returned error can't find the container with id 74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485
Nov 03 20:52:21.385242 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:21.385098    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerStarted","Data":"74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485"}
Nov 03 20:52:23.223064 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.221773    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"]
Nov 03 20:52:23.226664 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.226136    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.273482 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.273448    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 20:52:23.277412 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.276690    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.318992 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.318965    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7sxz2\""
Nov 03 20:52:23.320340 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.320302    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"]
Nov 03 20:52:23.328004 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.327981    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:52:23.328113 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.328084    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:52:23.358610 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358576    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.358776 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358623    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.358851 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358784    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.358851 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358812    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.358956 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358847    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.358956 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358901    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkz9\" (UniqueName: \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.358956 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358934    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.359106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.358965    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vwl\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.359106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.359003    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.359106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.359031    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.359106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.359082    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.359303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.359112    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.365630 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.365608    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:52:23.366516 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.366478    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 20:52:23.376214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.376195    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\""
Nov 03 20:52:23.459984 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.459929    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.459984 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.459975    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460007    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460031    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460060    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460107    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkz9\" (UniqueName: \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460144    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460171    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vwl\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460219 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460208    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460233    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460278    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460308    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.460849 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.460760    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.461217 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.461168    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.461617 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.461593    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.462393 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.462368    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.462492 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.462402    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.462556 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.462524    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.462689 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.462606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.463547 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.463505    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.467885 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.467812    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.540002 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.539465    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vwl\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.540002 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.539924    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-hb5wh\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.541916 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.541879    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkz9\" (UniqueName: \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.547483 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.547462    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:23.593423 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.593247    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:23.934156 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:23.934080    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"]
Nov 03 20:52:23.940929 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:23.940902    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98a0bdd_9f87_499e_b6f9_28d0a83e3b4f.slice/crio-95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301 WatchSource:0}: Error finding container 95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301: Status 404 returned error can't find the container with id 95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301
Nov 03 20:52:24.022858 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:24.022825    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 20:52:24.029097 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:24.028969    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4cbc95_15ee_4a79_a0a0_8887d4bf4bc3.slice/crio-96c097d5d6b4d9ba70ac3712d29340a66f0b0876fee5afaa39aea028f139bd4e WatchSource:0}: Error finding container 96c097d5d6b4d9ba70ac3712d29340a66f0b0876fee5afaa39aea028f139bd4e: Status 404 returned error can't find the container with id 96c097d5d6b4d9ba70ac3712d29340a66f0b0876fee5afaa39aea028f139bd4e
Nov 03 20:52:24.402658 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:24.402560    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerStarted","Data":"95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301"}
Nov 03 20:52:24.403671 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:24.403646    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerStarted","Data":"96c097d5d6b4d9ba70ac3712d29340a66f0b0876fee5afaa39aea028f139bd4e"}
Nov 03 20:52:26.140916 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.140881    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"]
Nov 03 20:52:26.145516 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.144977    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.173575 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.173552    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"]
Nov 03 20:52:26.177463 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.177120    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.220819 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.220751    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"]
Nov 03 20:52:26.250709 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.250649    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"]
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285612    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285685    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjb4\" (UniqueName: \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285720    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285796    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285821    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txr4\" (UniqueName: \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.286046 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.285858    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.387760    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.387842    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.388056    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjb4\" (UniqueName: \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.388098    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.388185    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.390652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.388418    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txr4\" (UniqueName: \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.392111 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.392047    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.392876 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.392382    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.392876 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.392666    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.393257 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.393232    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.455915 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.455858    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjb4\" (UniqueName: \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.458058 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.457972    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txr4\" (UniqueName: \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.466373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.466158    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:26.498619 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.498444    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:26.914439 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.914402    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"]
Nov 03 20:52:26.921480 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:26.921455    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"]
Nov 03 20:52:26.922388 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:26.922343    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9becc1_8bb0_441c_85ad_9e333517fd46.slice/crio-449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd WatchSource:0}: Error finding container 449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd: Status 404 returned error can't find the container with id 449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd
Nov 03 20:52:26.961672 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:26.961617    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1a2d9b_3c86_4df2_9e74_44e717c1d1d0.slice/crio-f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c WatchSource:0}: Error finding container f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c: Status 404 returned error can't find the container with id f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c
Nov 03 20:52:27.428684 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:27.428623    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48" event={"ID":"de9becc1-8bb0-441c-85ad-9e333517fd46","Type":"ContainerStarted","Data":"449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd"}
Nov 03 20:52:27.430823 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:27.430790    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerStarted","Data":"f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c"}
Nov 03 20:52:34.498276 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:34.498183    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerStarted","Data":"f13a41bd59924e67946da9ce6384c58d7eda40c258dc5fd9ba82dd8f479052d0"}
Nov 03 20:52:34.501922 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:34.501880    5533 generic.go:356] "Generic (PLEG): container finished" podID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerID="6898a5f6876c35d0f8ac4d8ebd1586b5eff2cb308176c85882c658276d1a0cd8" exitCode=0
Nov 03 20:52:34.502036 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:34.501924    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerDied","Data":"6898a5f6876c35d0f8ac4d8ebd1586b5eff2cb308176c85882c658276d1a0cd8"}
Nov 03 20:52:35.508719 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.508689    5533 generic.go:356] "Generic (PLEG): container finished" podID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerID="23806b28ffa4f0c23fa6f0b8c7b185558ea7195945af9505fa0f7906db2cc4d1" exitCode=0
Nov 03 20:52:35.509143 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.508774    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh" event={"ID":"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f","Type":"ContainerDied","Data":"23806b28ffa4f0c23fa6f0b8c7b185558ea7195945af9505fa0f7906db2cc4d1"}
Nov 03 20:52:35.512816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.512128    5533 generic.go:356] "Generic (PLEG): container finished" podID="bad1ea12-59b1-4471-adee-979d8f15802e" containerID="ba15e29e632d3f35be56d4e6eeed43a80cddc7653673b9166bdee1a540a325a0" exitCode=0
Nov 03 20:52:35.512816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.512204    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerDied","Data":"ba15e29e632d3f35be56d4e6eeed43a80cddc7653673b9166bdee1a540a325a0"}
Nov 03 20:52:35.516976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.516280    5533 generic.go:356] "Generic (PLEG): container finished" podID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerID="ab18aed4ad79410d6509d2e4b5246e1b3b7046ed3752f74c16fe261d7795aa4e" exitCode=0
Nov 03 20:52:35.516976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.516394    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf" event={"ID":"83bcf0cc-19d3-4896-b41a-d5e8eee25017","Type":"ContainerDied","Data":"ab18aed4ad79410d6509d2e4b5246e1b3b7046ed3752f74c16fe261d7795aa4e"}
Nov 03 20:52:35.521605 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.521012    5533 generic.go:356] "Generic (PLEG): container finished" podID="9961fc23-444d-4279-b442-ca20168d2a56" containerID="f13a41bd59924e67946da9ce6384c58d7eda40c258dc5fd9ba82dd8f479052d0" exitCode=0
Nov 03 20:52:35.521605 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.521066    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerDied","Data":"f13a41bd59924e67946da9ce6384c58d7eda40c258dc5fd9ba82dd8f479052d0"}
Nov 03 20:52:35.661233 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.660780    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:35.661233 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.660853    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:35.661233 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:35.660897    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:36.528023 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.527935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerStarted","Data":"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"}
Nov 03 20:52:36.530585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.530554    5533 generic.go:356] "Generic (PLEG): container finished" podID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerID="98963803ea815225c8f6ba6cf265fc116a22a81a58091c0cdfe7f5295ac25005" exitCode=0
Nov 03 20:52:36.530747 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.530654    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerDied","Data":"98963803ea815225c8f6ba6cf265fc116a22a81a58091c0cdfe7f5295ac25005"}
Nov 03 20:52:36.532844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.532812    5533 generic.go:356] "Generic (PLEG): container finished" podID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerID="ba52fd7608112a4743827968461d50e483637e0af0c607b64a9753667983d278" exitCode=0
Nov 03 20:52:36.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.532897    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48" event={"ID":"de9becc1-8bb0-441c-85ad-9e333517fd46","Type":"ContainerDied","Data":"ba52fd7608112a4743827968461d50e483637e0af0c607b64a9753667983d278"}
Nov 03 20:52:36.593567 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.593542    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:36.599168 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.599141    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:36.643823 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:36.643577    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podStartSLOduration=2.015754349 podStartE2EDuration="13.643561195s" totalImagesPullingTime="11.627806846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:36.641610712 +0000 UTC m=+258.272924210" watchObservedRunningTime="2026-11-03 20:52:36.643561195 +0000 UTC m=+258.274874674"
Nov 03 20:52:37.538017 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.537991    5533 generic.go:356] "Generic (PLEG): container finished" podID="9961fc23-444d-4279-b442-ca20168d2a56" containerID="99f74bb5a5290b109c459803ac2fe7514cfd4d331ce7ea8db213d3fdde17bd6a" exitCode=0
Nov 03 20:52:37.538394 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.538069    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerDied","Data":"99f74bb5a5290b109c459803ac2fe7514cfd4d331ce7ea8db213d3fdde17bd6a"}
Nov 03 20:52:37.539697 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.539675    5533 generic.go:356] "Generic (PLEG): container finished" podID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerID="7eb5623901d7aa3b0103e3bdf92f6ffb1f81c4f9ac061a5c47bb8952bb8e8dfd" exitCode=0
Nov 03 20:52:37.539817 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.539737    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerDied","Data":"7eb5623901d7aa3b0103e3bdf92f6ffb1f81c4f9ac061a5c47bb8952bb8e8dfd"}
Nov 03 20:52:37.539952 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.539935    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:37.541048 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:37.541033    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:52:38.544571 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.544544    5533 generic.go:356] "Generic (PLEG): container finished" podID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerID="9bf79ea96be875896bea025a827804b1eed5a114e45c6b15695bdeea8173923a" exitCode=0
Nov 03 20:52:38.544930 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.544615    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh" event={"ID":"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f","Type":"ContainerDied","Data":"9bf79ea96be875896bea025a827804b1eed5a114e45c6b15695bdeea8173923a"}
Nov 03 20:52:38.546454 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.546431    5533 generic.go:356] "Generic (PLEG): container finished" podID="bad1ea12-59b1-4471-adee-979d8f15802e" containerID="d71c68b46940a86be1c035b00aa952bedea6dc34194d652f707467bb08a98460" exitCode=0
Nov 03 20:52:38.546517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.546504    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerDied","Data":"d71c68b46940a86be1c035b00aa952bedea6dc34194d652f707467bb08a98460"}
Nov 03 20:52:38.548362 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.548341    5533 generic.go:356] "Generic (PLEG): container finished" podID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerID="a5e65cefe8e160d76eff2d2dae71062e158b85728e1d11cdd3a53e3803fab345" exitCode=0
Nov 03 20:52:38.548435 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.548413    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf" event={"ID":"83bcf0cc-19d3-4896-b41a-d5e8eee25017","Type":"ContainerDied","Data":"a5e65cefe8e160d76eff2d2dae71062e158b85728e1d11cdd3a53e3803fab345"}
Nov 03 20:52:38.549972 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.549951    5533 generic.go:356] "Generic (PLEG): container finished" podID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerID="04d1b76e9fbf86b4e8093d9129abfed39176da8d22fdee6d367cd4a8ac184326" exitCode=0
Nov 03 20:52:38.550061 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.550033    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerDied","Data":"04d1b76e9fbf86b4e8093d9129abfed39176da8d22fdee6d367cd4a8ac184326"}
Nov 03 20:52:38.551999 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.551914    5533 generic.go:356] "Generic (PLEG): container finished" podID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerID="5ce41d054585f4457366a64eece2a406b086b7c469df573de6e2bfa3ec30d46c" exitCode=0
Nov 03 20:52:38.551999 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:38.551979    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48" event={"ID":"de9becc1-8bb0-441c-85ad-9e333517fd46","Type":"ContainerDied","Data":"5ce41d054585f4457366a64eece2a406b086b7c469df573de6e2bfa3ec30d46c"}
Nov 03 20:52:39.558792 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:39.558534    5533 generic.go:356] "Generic (PLEG): container finished" podID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerID="07f6756036bf2bf7d7ffc99d8e32aae82bbd101f0f0f38acf583dc2f7790f41b" exitCode=0
Nov 03 20:52:39.558792 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:39.558597    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerDied","Data":"07f6756036bf2bf7d7ffc99d8e32aae82bbd101f0f0f38acf583dc2f7790f41b"}
Nov 03 20:52:40.566517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:40.566460    5533 generic.go:356] "Generic (PLEG): container finished" podID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerID="f62e14d5d1f98ae0a9c4a71e570795619b19edbe920262d8a35484297a8c5c2d" exitCode=0
Nov 03 20:52:40.566989 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:40.566522    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerDied","Data":"f62e14d5d1f98ae0a9c4a71e570795619b19edbe920262d8a35484297a8c5c2d"}
Nov 03 20:52:43.586692 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:43.586603    5533 generic.go:356] "Generic (PLEG): container finished" podID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerID="80f68debfcc10505ad98fd060c5801a3dc0d7f4f3bc0c6fb2054b0320f43b0f4" exitCode=0
Nov 03 20:52:43.587066 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:43.586764    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerDied","Data":"80f68debfcc10505ad98fd060c5801a3dc0d7f4f3bc0c6fb2054b0320f43b0f4"}
Nov 03 20:52:44.592891 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:44.592816    5533 generic.go:356] "Generic (PLEG): container finished" podID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerID="5be0c54d5b982a4e174c00f8fae3fa593df445c0a3e57e91b7b80f0976ff4aa9" exitCode=0
Nov 03 20:52:44.592891 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:44.592871    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerDied","Data":"5be0c54d5b982a4e174c00f8fae3fa593df445c0a3e57e91b7b80f0976ff4aa9"}
Nov 03 20:52:49.295927 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:52:49.295888    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:52:50.643602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:50.643565    5533 generic.go:356] "Generic (PLEG): container finished" podID="9961fc23-444d-4279-b442-ca20168d2a56" containerID="86811dde7468615782a7ce57e831d05da7e62d8f790d1da11a693a9748f8a183" exitCode=0
Nov 03 20:52:50.644143 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:50.644114    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerDied","Data":"86811dde7468615782a7ce57e831d05da7e62d8f790d1da11a693a9748f8a183"}
Nov 03 20:52:50.648190 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:50.648162    5533 generic.go:356] "Generic (PLEG): container finished" podID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerID="8b8afa4bacc1c27ea0eef92abbc90b36e6c1bcc5f33bc4919756970f409be1cf" exitCode=0
Nov 03 20:52:50.648303 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:50.648262    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh" event={"ID":"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f","Type":"ContainerDied","Data":"8b8afa4bacc1c27ea0eef92abbc90b36e6c1bcc5f33bc4919756970f409be1cf"}
Nov 03 20:52:51.657436 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.657395    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerStarted","Data":"6761f4919769b1d3d128433cde39a41d9043cf7be81b70f1c6f54bc2ca4eec61"}
Nov 03 20:52:51.783868 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.783286    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" podStartSLOduration=2.680094008 podStartE2EDuration="31.783267895s" totalImagesPullingTime="29.103173887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:51.782471831 +0000 UTC m=+273.413785307" watchObservedRunningTime="2026-11-03 20:52:51.783267895 +0000 UTC m=+273.414581373"
Nov 03 20:52:51.870249 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.869973    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:51.874101 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.874078    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:51.947343 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947304    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\") pod \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") "
Nov 03 20:52:51.947502 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947390    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\" (UniqueName: \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\") pod \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") "
Nov 03 20:52:51.947502 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947424    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\") pod \"9961fc23-444d-4279-b442-ca20168d2a56\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") "
Nov 03 20:52:51.947502 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947459    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\" (UniqueName: \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\") pod \"9961fc23-444d-4279-b442-ca20168d2a56\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") "
Nov 03 20:52:51.947502 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947489    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\") pod \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\" (UID: \"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f\") "
Nov 03 20:52:51.947842 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.947541    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\") pod \"9961fc23-444d-4279-b442-ca20168d2a56\" (UID: \"9961fc23-444d-4279-b442-ca20168d2a56\") "
Nov 03 20:52:51.948955 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.948904    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle" pod "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" (UID: "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:51.955181 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.955153    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle" pod "9961fc23-444d-4279-b442-ca20168d2a56" (UID: "9961fc23-444d-4279-b442-ca20168d2a56"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:51.956505 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.956478    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util" pod "9961fc23-444d-4279-b442-ca20168d2a56" (UID: "9961fc23-444d-4279-b442-ca20168d2a56"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:51.961490 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.961459    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util" pod "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" (UID: "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:51.963785 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.963753    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888" pod "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" (UID: "404d2d4a-3b45-413a-9a44-b2ec6bddcf3f"). InnerVolumeSpecName "kube-api-access-lv888". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:51.969876 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:51.969849    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc" pod "9961fc23-444d-4279-b442-ca20168d2a56" (UID: "9961fc23-444d-4279-b442-ca20168d2a56"). InnerVolumeSpecName "kube-api-access-9fxdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:52.048169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048096    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv888\" (UniqueName: \"kubernetes.io/projected/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-kube-api-access-lv888\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.048169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048131    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.048169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048147    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxdc\" (UniqueName: \"kubernetes.io/projected/9961fc23-444d-4279-b442-ca20168d2a56-kube-api-access-9fxdc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.048169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048161    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.048602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048177    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9961fc23-444d-4279-b442-ca20168d2a56-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.048602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.048190    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404d2d4a-3b45-413a-9a44-b2ec6bddcf3f-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:52.666038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.666005    5533 generic.go:356] "Generic (PLEG): container finished" podID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerID="6761f4919769b1d3d128433cde39a41d9043cf7be81b70f1c6f54bc2ca4eec61" exitCode=0
Nov 03 20:52:52.666571 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.666092    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerDied","Data":"6761f4919769b1d3d128433cde39a41d9043cf7be81b70f1c6f54bc2ca4eec61"}
Nov 03 20:52:52.669505 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.668740    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs" event={"ID":"9961fc23-444d-4279-b442-ca20168d2a56","Type":"ContainerDied","Data":"521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23"}
Nov 03 20:52:52.669505 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.668786    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="521c57fd88414bc3a9f859feac7bd1358b3f529bc9ca795c7e969eb2a2940c23"
Nov 03 20:52:52.669505 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.668871    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gvmbs"
Nov 03 20:52:52.675666 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.675512    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh" event={"ID":"404d2d4a-3b45-413a-9a44-b2ec6bddcf3f","Type":"ContainerDied","Data":"170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a"}
Nov 03 20:52:52.675666 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.675551    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170d9e4bddf89ea0e451702f9e54cc9ee2ea7fb2231e814b0633b84dfcb5555a"
Nov 03 20:52:52.675666 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:52.675618    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d776nzh"
Nov 03 20:52:53.683377 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.683290    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerStarted","Data":"c150b9f59f150b60de98b1e8bae7fa07cad8caca528ddef0744d0edfaf9471d8"}
Nov 03 20:52:53.687328 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.687293    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerStarted","Data":"d852ac9e5304d507a5703cdf362b67d0a37473c12f5fde60c194ae2d65fc8155"}
Nov 03 20:52:53.795800 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.795238    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" podStartSLOduration=2.734860818 podStartE2EDuration="30.795221214s" totalImagesPullingTime="28.060360396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:53.794070801 +0000 UTC m=+275.425384280" watchObservedRunningTime="2026-11-03 20:52:53.795221214 +0000 UTC m=+275.426534688"
Nov 03 20:52:53.906081 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.906022    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" podStartSLOduration=2.438957686 podStartE2EDuration="27.906003088s" totalImagesPullingTime="25.467045402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:53.904456308 +0000 UTC m=+275.535769786" watchObservedRunningTime="2026-11-03 20:52:53.906003088 +0000 UTC m=+275.537316569"
Nov 03 20:52:53.934730 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.934707    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:53.975530 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.975502    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\") pod \"09523415-47a1-437b-b5c5-b8a4350fed2e\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") "
Nov 03 20:52:53.975648 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.975571    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\") pod \"09523415-47a1-437b-b5c5-b8a4350fed2e\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") "
Nov 03 20:52:53.975648 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.975615    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\" (UniqueName: \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\") pod \"09523415-47a1-437b-b5c5-b8a4350fed2e\" (UID: \"09523415-47a1-437b-b5c5-b8a4350fed2e\") "
Nov 03 20:52:53.980898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.980857    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle" pod "09523415-47a1-437b-b5c5-b8a4350fed2e" (UID: "09523415-47a1-437b-b5c5-b8a4350fed2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:53.985723 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.985683    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util" pod "09523415-47a1-437b-b5c5-b8a4350fed2e" (UID: "09523415-47a1-437b-b5c5-b8a4350fed2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:53.991508 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:53.991484    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx" pod "09523415-47a1-437b-b5c5-b8a4350fed2e" (UID: "09523415-47a1-437b-b5c5-b8a4350fed2e"). InnerVolumeSpecName "kube-api-access-xk6tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:54.077313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.077269    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.077313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.077297    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09523415-47a1-437b-b5c5-b8a4350fed2e-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.077313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.077312    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6tx\" (UniqueName: \"kubernetes.io/projected/09523415-47a1-437b-b5c5-b8a4350fed2e-kube-api-access-xk6tx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:54.692330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.692306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr" event={"ID":"09523415-47a1-437b-b5c5-b8a4350fed2e","Type":"ContainerDied","Data":"74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485"}
Nov 03 20:52:54.692330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.692321    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tlspr"
Nov 03 20:52:54.692748 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.692349    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c16a29440d3d5af27c451716f12f898150f616bedd23e7414cf758f11d0485"
Nov 03 20:52:54.694660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.694624    5533 generic.go:356] "Generic (PLEG): container finished" podID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerID="d852ac9e5304d507a5703cdf362b67d0a37473c12f5fde60c194ae2d65fc8155" exitCode=0
Nov 03 20:52:54.694775 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.694725    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerDied","Data":"d852ac9e5304d507a5703cdf362b67d0a37473c12f5fde60c194ae2d65fc8155"}
Nov 03 20:52:54.696681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.696660    5533 generic.go:356] "Generic (PLEG): container finished" podID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerID="85c8c8649b0d27c3dcaa95e1821a951156be916c7fc7a7d6ce2264e25747ecb0" exitCode=0
Nov 03 20:52:54.696765 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.696738    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48" event={"ID":"de9becc1-8bb0-441c-85ad-9e333517fd46","Type":"ContainerDied","Data":"85c8c8649b0d27c3dcaa95e1821a951156be916c7fc7a7d6ce2264e25747ecb0"}
Nov 03 20:52:54.698588 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.698571    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerStarted","Data":"cf37483fc95a1e9854b107e6896b275fca8806dd5fdebd7b15c0296ccb7d8b1b"}
Nov 03 20:52:54.700541 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.700523    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerStarted","Data":"8a0a9ea262e449d78ef910953a96814f23633cc0304dd5eda3a03f6a94fee109"}
Nov 03 20:52:54.702254 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.702237    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:52:54.702975 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.702956    5533 generic.go:356] "Generic (PLEG): container finished" podID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerID="25a38ec58bc913767e432b0b6b39e77acdf633bd558fd7f5af79775f5d3aa856" exitCode=1
Nov 03 20:52:54.703085 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.702989    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf" event={"ID":"83bcf0cc-19d3-4896-b41a-d5e8eee25017","Type":"ContainerDied","Data":"25a38ec58bc913767e432b0b6b39e77acdf633bd558fd7f5af79775f5d3aa856"}
Nov 03 20:52:54.705057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.705038    5533 generic.go:356] "Generic (PLEG): container finished" podID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerID="c150b9f59f150b60de98b1e8bae7fa07cad8caca528ddef0744d0edfaf9471d8" exitCode=0
Nov 03 20:52:54.705140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:54.705063    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerDied","Data":"c150b9f59f150b60de98b1e8bae7fa07cad8caca528ddef0744d0edfaf9471d8"}
Nov 03 20:52:55.079668 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.079576    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" podStartSLOduration=3.072601647 podStartE2EDuration="37.079562341s" totalImagesPullingTime="34.006960694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:55.078834978 +0000 UTC m=+276.710148445" watchObservedRunningTime="2026-11-03 20:52:55.079562341 +0000 UTC m=+276.710875818"
Nov 03 20:52:55.300779 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.300734    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" podStartSLOduration=3.8309989939999998 podStartE2EDuration="37.300721581s" totalImagesPullingTime="33.469722587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:55.300323582 +0000 UTC m=+276.931637059" watchObservedRunningTime="2026-11-03 20:52:55.300721581 +0000 UTC m=+276.932035096"
Nov 03 20:52:55.711470 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.711434    5533 generic.go:356] "Generic (PLEG): container finished" podID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerID="cf37483fc95a1e9854b107e6896b275fca8806dd5fdebd7b15c0296ccb7d8b1b" exitCode=0
Nov 03 20:52:55.711872 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.711514    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerDied","Data":"cf37483fc95a1e9854b107e6896b275fca8806dd5fdebd7b15c0296ccb7d8b1b"}
Nov 03 20:52:55.713671 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.713647    5533 generic.go:356] "Generic (PLEG): container finished" podID="bad1ea12-59b1-4471-adee-979d8f15802e" containerID="8a0a9ea262e449d78ef910953a96814f23633cc0304dd5eda3a03f6a94fee109" exitCode=0
Nov 03 20:52:55.713773 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.713673    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerDied","Data":"8a0a9ea262e449d78ef910953a96814f23633cc0304dd5eda3a03f6a94fee109"}
Nov 03 20:52:55.921955 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.921935    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:52:55.939783 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.939764    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:55.959261 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.959241    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:55.973835 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.973786    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:55.987301 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.987282    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:55.990264 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990246    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\") pod \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") "
Nov 03 20:52:55.990309 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990291    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\") pod \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") "
Nov 03 20:52:55.990353 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990334    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\") pod \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") "
Nov 03 20:52:55.990384 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990371    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\" (UniqueName: \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\") pod \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") "
Nov 03 20:52:55.990434 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990421    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\" (UniqueName: \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\") pod \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\" (UID: \"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0\") "
Nov 03 20:52:55.990496 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990481    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\") pod \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\" (UID: \"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f\") "
Nov 03 20:52:55.990985 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.990958    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle" pod "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" (UID: "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:55.991075 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.991050    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle" pod "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" (UID: "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:55.992390 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.992373    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9" pod "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" (UID: "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f"). InnerVolumeSpecName "kube-api-access-2rkz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:55.992852 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.992830    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4" pod "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" (UID: "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0"). InnerVolumeSpecName "kube-api-access-7txr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:55.994593 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.994575    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util" pod "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" (UID: "f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:55.995161 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:55.995141    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util" pod "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" (UID: "7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.091763 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091739    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\") pod \"de9becc1-8bb0-441c-85ad-9e333517fd46\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") "
Nov 03 20:52:56.091898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091783    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\") pod \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") "
Nov 03 20:52:56.091898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091801    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\" (UniqueName: \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\") pod \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") "
Nov 03 20:52:56.091898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091829    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\" (UniqueName: \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\") pod \"de9becc1-8bb0-441c-85ad-9e333517fd46\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") "
Nov 03 20:52:56.091898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091857    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\") pod \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\" (UID: \"83bcf0cc-19d3-4896-b41a-d5e8eee25017\") "
Nov 03 20:52:56.091898 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.091874    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\") pod \"de9becc1-8bb0-441c-85ad-9e333517fd46\" (UID: \"de9becc1-8bb0-441c-85ad-9e333517fd46\") "
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092050    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092070    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092082    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092095    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092109    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rkz9\" (UniqueName: \"kubernetes.io/projected/f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f-kube-api-access-2rkz9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092125    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txr4\" (UniqueName: \"kubernetes.io/projected/7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0-kube-api-access-7txr4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.092676 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092650    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle" pod "83bcf0cc-19d3-4896-b41a-d5e8eee25017" (UID: "83bcf0cc-19d3-4896-b41a-d5e8eee25017"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.092767 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.092689    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle" pod "de9becc1-8bb0-441c-85ad-9e333517fd46" (UID: "de9becc1-8bb0-441c-85ad-9e333517fd46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.093906 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.093869    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4" pod "de9becc1-8bb0-441c-85ad-9e333517fd46" (UID: "de9becc1-8bb0-441c-85ad-9e333517fd46"). InnerVolumeSpecName "kube-api-access-8vjb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:56.094129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.094113    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv" pod "83bcf0cc-19d3-4896-b41a-d5e8eee25017" (UID: "83bcf0cc-19d3-4896-b41a-d5e8eee25017"). InnerVolumeSpecName "kube-api-access-cwprv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:56.096724 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.096699    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util" pod "de9becc1-8bb0-441c-85ad-9e333517fd46" (UID: "de9becc1-8bb0-441c-85ad-9e333517fd46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.098055 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.098032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util" pod "83bcf0cc-19d3-4896-b41a-d5e8eee25017" (UID: "83bcf0cc-19d3-4896-b41a-d5e8eee25017"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.193286 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193264    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.193373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193293    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.193373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193303    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwprv\" (UniqueName: \"kubernetes.io/projected/83bcf0cc-19d3-4896-b41a-d5e8eee25017-kube-api-access-cwprv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.193373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193312    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjb4\" (UniqueName: \"kubernetes.io/projected/de9becc1-8bb0-441c-85ad-9e333517fd46-kube-api-access-8vjb4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.193373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193321    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bcf0cc-19d3-4896-b41a-d5e8eee25017-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.193373 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.193328    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de9becc1-8bb0-441c-85ad-9e333517fd46-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:56.719487 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.719460    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5"
Nov 03 20:52:56.719847 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.719459    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jn5g5" event={"ID":"7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0","Type":"ContainerDied","Data":"f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c"}
Nov 03 20:52:56.719847 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.719596    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fc07d407653204c6e3407161d7b4b360f7af8a12e3a8c2a218e9ec9e2ec35c"
Nov 03 20:52:56.721508 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.721490    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48"
Nov 03 20:52:56.721629 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.721523    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amgj48" event={"ID":"de9becc1-8bb0-441c-85ad-9e333517fd46","Type":"ContainerDied","Data":"449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd"}
Nov 03 20:52:56.721629 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.721565    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449c7ad0217954c9681e999df583268d85f3665e62ab60a4288676a4202d78bd"
Nov 03 20:52:56.722941 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.722922    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:52:56.723804 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.723782    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf" event={"ID":"83bcf0cc-19d3-4896-b41a-d5e8eee25017","Type":"ContainerDied","Data":"3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f"}
Nov 03 20:52:56.723886 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.723817    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eab2e4959ccb20b0c49b31df930c7c6f78403021226a180810d7c84d489307f"
Nov 03 20:52:56.723886 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.723785    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf"
Nov 03 20:52:56.725824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.725803    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv" event={"ID":"f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f","Type":"ContainerDied","Data":"95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301"}
Nov 03 20:52:56.725918 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.725838    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qqgbv"
Nov 03 20:52:56.725918 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.725840    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95e7f12e195fb26d0624914debca98ce5549f0ae77f7082ed7b8328f3bb60301"
Nov 03 20:52:56.872748 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.872724    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:56.884526 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.884506    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:56.997483 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997458    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\" (UniqueName: \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\") pod \"bad1ea12-59b1-4471-adee-979d8f15802e\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") "
Nov 03 20:52:56.997588 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997498    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\") pod \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") "
Nov 03 20:52:56.997588 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997517    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\") pod \"bad1ea12-59b1-4471-adee-979d8f15802e\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") "
Nov 03 20:52:56.997724 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997650    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\") pod \"bad1ea12-59b1-4471-adee-979d8f15802e\" (UID: \"bad1ea12-59b1-4471-adee-979d8f15802e\") "
Nov 03 20:52:56.997782 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997751    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\" (UniqueName: \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\") pod \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") "
Nov 03 20:52:56.997845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.997800    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\") pod \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\" (UID: \"badf81b0-3b0b-4b9b-ac77-fdb73213b676\") "
Nov 03 20:52:56.998265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.998235    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle" pod "bad1ea12-59b1-4471-adee-979d8f15802e" (UID: "bad1ea12-59b1-4471-adee-979d8f15802e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.998409 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.998382    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle" pod "badf81b0-3b0b-4b9b-ac77-fdb73213b676" (UID: "badf81b0-3b0b-4b9b-ac77-fdb73213b676"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:56.999937 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.999904    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh" pod "badf81b0-3b0b-4b9b-ac77-fdb73213b676" (UID: "badf81b0-3b0b-4b9b-ac77-fdb73213b676"). InnerVolumeSpecName "kube-api-access-2c7rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:57.000014 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:56.999939    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7" pod "bad1ea12-59b1-4471-adee-979d8f15802e" (UID: "bad1ea12-59b1-4471-adee-979d8f15802e"). InnerVolumeSpecName "kube-api-access-ln4f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:57.003110 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.003013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util" pod "bad1ea12-59b1-4471-adee-979d8f15802e" (UID: "bad1ea12-59b1-4471-adee-979d8f15802e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:57.004227 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.004204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util" pod "badf81b0-3b0b-4b9b-ac77-fdb73213b676" (UID: "badf81b0-3b0b-4b9b-ac77-fdb73213b676"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:57.098667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098589    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7rh\" (UniqueName: \"kubernetes.io/projected/badf81b0-3b0b-4b9b-ac77-fdb73213b676-kube-api-access-2c7rh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.098667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098612    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.098667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098626    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4f7\" (UniqueName: \"kubernetes.io/projected/bad1ea12-59b1-4471-adee-979d8f15802e-kube-api-access-ln4f7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.098667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098651    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/badf81b0-3b0b-4b9b-ac77-fdb73213b676-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.098667 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098664    5533 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-bundle\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.098864 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.098677    5533 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bad1ea12-59b1-4471-adee-979d8f15802e-util\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:57.732678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.732651    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq"
Nov 03 20:52:57.733130 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.732632    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb87rq" event={"ID":"badf81b0-3b0b-4b9b-ac77-fdb73213b676","Type":"ContainerDied","Data":"5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e"}
Nov 03 20:52:57.733130 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.732791    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5030b4b658d23bddabbeff729740f4a0d5a0281896f5d89362ec17087b139e"
Nov 03 20:52:57.734597 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.734575    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg"
Nov 03 20:52:57.734724 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.734574    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj8dbg" event={"ID":"bad1ea12-59b1-4471-adee-979d8f15802e","Type":"ContainerDied","Data":"4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f"}
Nov 03 20:52:57.734724 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:57.734687    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4235099cd27390512f79c2e7c93097be1fe794bbc62df001921335486e19f18f"
Nov 03 20:52:58.531165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531133    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"]
Nov 03 20:52:58.531390 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531377    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="util"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531392    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="util"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531404    5533 cpu_manager.go:414] "containerMap: removing container" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="pull"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531409    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="pull"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531418    5533 cpu_manager.go:414] "containerMap: removing container" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="pull"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531423    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="pull"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531431    5533 cpu_manager.go:414] "containerMap: removing container" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="extract"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531435    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="extract"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531441    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="extract"
Nov 03 20:52:58.531445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531446    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531453    5533 cpu_manager.go:414] "containerMap: removing container" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531458    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531465    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531470    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531476    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531480    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531487    5533 cpu_manager.go:414] "containerMap: removing container" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531490    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531496    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531500    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531508    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531512    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531518    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531522    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531527    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531531    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531539    5533 cpu_manager.go:414] "containerMap: removing container" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531545    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531552    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531556    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531562    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531566    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531572    5533 cpu_manager.go:414] "containerMap: removing container" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531577    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531584    5533 cpu_manager.go:414] "containerMap: removing container" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531588    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531595    5533 cpu_manager.go:414] "containerMap: removing container" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531599    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531604    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531608    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531613    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531618    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531627    5533 cpu_manager.go:414] "containerMap: removing container" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531644    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="extract"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531654    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531658    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531665    5533 cpu_manager.go:414] "containerMap: removing container" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531669    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531677    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531681    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531687    5533 cpu_manager.go:414] "containerMap: removing container" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531691    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="pull"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531697    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="util"
Nov 03 20:52:58.531720 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531702    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="util"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531747    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad1ea12-59b1-4471-adee-979d8f15802e" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531755    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9becc1-8bb0-441c-85ad-9e333517fd46" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531761    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="09523415-47a1-437b-b5c5-b8a4350fed2e" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531766    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="404d2d4a-3b45-413a-9a44-b2ec6bddcf3f" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531770    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1a2d9b-3c86-4df2-9e74-44e717c1d1d0" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531776    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9961fc23-444d-4279-b442-ca20168d2a56" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531781    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="badf81b0-3b0b-4b9b-ac77-fdb73213b676" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531787    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98a0bdd-9f87-499e-b6f9-28d0a83e3b4f" containerName="extract"
Nov 03 20:52:58.532951 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.531792    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bcf0cc-19d3-4896-b41a-d5e8eee25017" containerName="extract"
Nov 03 20:52:58.560256 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.560232    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.589545 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.589520    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"]
Nov 03 20:52:58.609675 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.609646    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-tmp\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.609761 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.609720    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr26m\" (UniqueName: \"kubernetes.io/projected/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-kube-api-access-tr26m\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.634891 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.634873    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:52:58.634954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.634915    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 20:52:58.634990 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.634966    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fwsf8\""
Nov 03 20:52:58.710068 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.710049    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-tmp\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.710178 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.710084    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr26m\" (UniqueName: \"kubernetes.io/projected/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-kube-api-access-tr26m\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.710375 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.710357    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-tmp\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.767555 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.767532    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr26m\" (UniqueName: \"kubernetes.io/projected/28b3a8a5-3857-45b3-aebc-caeae5cbfa53-kube-api-access-tr26m\") pod \"jobset-operator-674cd59dc5-j8pbv\" (UID: \"28b3a8a5-3857-45b3-aebc-caeae5cbfa53\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:58.901484 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:58.901433    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"
Nov 03 20:52:59.276071 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:59.276043    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv"]
Nov 03 20:52:59.279185 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:52:59.279157    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b3a8a5_3857_45b3_aebc_caeae5cbfa53.slice/crio-56d82879ffd68c6613326e3d2136e872059e23626761ce395cd808638ce9912c WatchSource:0}: Error finding container 56d82879ffd68c6613326e3d2136e872059e23626761ce395cd808638ce9912c: Status 404 returned error can't find the container with id 56d82879ffd68c6613326e3d2136e872059e23626761ce395cd808638ce9912c
Nov 03 20:52:59.742958 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:52:59.742926    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv" event={"ID":"28b3a8a5-3857-45b3-aebc-caeae5cbfa53","Type":"ContainerStarted","Data":"56d82879ffd68c6613326e3d2136e872059e23626761ce395cd808638ce9912c"}
Nov 03 20:53:04.765812 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:04.765776    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv" event={"ID":"28b3a8a5-3857-45b3-aebc-caeae5cbfa53","Type":"ContainerStarted","Data":"3615de12fbeb9d8f9fe5954cea4c94141105c4d85a3a477ade3f151232c8ed98"}
Nov 03 20:53:04.875223 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:04.875176    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j8pbv" podStartSLOduration=2.002062044 podStartE2EDuration="6.875163647s" totalImagesPullingTime="4.873101603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:04.874542724 +0000 UTC m=+286.505856201" watchObservedRunningTime="2026-11-03 20:53:04.875163647 +0000 UTC m=+286.506477125"
Nov 03 20:53:09.918351 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.918316    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"]
Nov 03 20:53:09.921746 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.921730    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.980928 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.980900    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.981056 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.980939    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-kube-api-access-xhjtl\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.981056 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.980960    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.981056 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.981019    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.981158 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.981101    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:09.981195 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:09.981156    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.081914 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.081882    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.081914 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.081915    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.082092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.081934    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.082092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.081960    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.082092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.081984    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.082092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.082009    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-kube-api-access-xhjtl\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.082332 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.082305    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-manifests\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.083984 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.083965    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-tmp\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.094020 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.093993    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:53:10.094140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.094035    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:53:10.094140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.094074    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:53:10.094140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.094096    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:53:10.094140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.094111    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pc7cb\""
Nov 03 20:53:10.094345 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.094142    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:53:10.104376 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.104355    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-metrics-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.104376 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.104366    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.104514 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.104491    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.139522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.139494    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjtl\" (UniqueName: \"kubernetes.io/projected/d8faa8b2-ba59-46b1-8f4c-1a043ac3a595-kube-api-access-xhjtl\") pod \"opendatahub-operator-controller-manager-6ffb58b49d-qvqnv\" (UID: \"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.153228 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.153203    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"]
Nov 03 20:53:10.237735 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.237711    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:10.717780 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:10.717756    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8faa8b2_ba59_46b1_8f4c_1a043ac3a595.slice/crio-25d4e98fceb5dc0812bb22075c758305e4f8e72fa3541d67fa2b59b616797afe WatchSource:0}: Error finding container 25d4e98fceb5dc0812bb22075c758305e4f8e72fa3541d67fa2b59b616797afe: Status 404 returned error can't find the container with id 25d4e98fceb5dc0812bb22075c758305e4f8e72fa3541d67fa2b59b616797afe
Nov 03 20:53:10.761025 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.760998    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"]
Nov 03 20:53:10.791436 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:10.791409    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv" event={"ID":"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595","Type":"ContainerStarted","Data":"25d4e98fceb5dc0812bb22075c758305e4f8e72fa3541d67fa2b59b616797afe"}
Nov 03 20:53:14.560506 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.560479    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"]
Nov 03 20:53:14.565082 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.565067    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:14.619579 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.619553    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s68l\" (UniqueName: \"kubernetes.io/projected/128dab95-58a5-4fc4-bc64-21b427628fc2-kube-api-access-9s68l\") pod \"limitador-operator-controller-manager-5778bb4449-ngx24\" (UID: \"128dab95-58a5-4fc4-bc64-21b427628fc2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:14.660845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.660818    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x28mk\""
Nov 03 20:53:14.660845 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.660838    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:53:14.660967 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.660860    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:53:14.662075 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.662058    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"]
Nov 03 20:53:14.719911 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.719887    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s68l\" (UniqueName: \"kubernetes.io/projected/128dab95-58a5-4fc4-bc64-21b427628fc2-kube-api-access-9s68l\") pod \"limitador-operator-controller-manager-5778bb4449-ngx24\" (UID: \"128dab95-58a5-4fc4-bc64-21b427628fc2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:14.783899 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.783881    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s68l\" (UniqueName: \"kubernetes.io/projected/128dab95-58a5-4fc4-bc64-21b427628fc2-kube-api-access-9s68l\") pod \"limitador-operator-controller-manager-5778bb4449-ngx24\" (UID: \"128dab95-58a5-4fc4-bc64-21b427628fc2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:14.812670 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.812613    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv" event={"ID":"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595","Type":"ContainerStarted","Data":"7a94651586216e9f2f0ee16be4412757587057f7cc3446aefd6e6d0cde3cefda"}
Nov 03 20:53:14.876611 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:14.876589    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:15.249386 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:15.249352    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"]
Nov 03 20:53:15.255567 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:15.255533    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128dab95_58a5_4fc4_bc64_21b427628fc2.slice/crio-d3b1ca024d3aebccfa91df1dcccfcffe05f3e2893c4cef0d6f45d4ee3356494f WatchSource:0}: Error finding container d3b1ca024d3aebccfa91df1dcccfcffe05f3e2893c4cef0d6f45d4ee3356494f: Status 404 returned error can't find the container with id d3b1ca024d3aebccfa91df1dcccfcffe05f3e2893c4cef0d6f45d4ee3356494f
Nov 03 20:53:15.819814 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:15.819781    5533 generic.go:356] "Generic (PLEG): container finished" podID="d8faa8b2-ba59-46b1-8f4c-1a043ac3a595" containerID="7a94651586216e9f2f0ee16be4412757587057f7cc3446aefd6e6d0cde3cefda" exitCode=0
Nov 03 20:53:15.820294 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:15.819873    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv" event={"ID":"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595","Type":"ContainerDied","Data":"7a94651586216e9f2f0ee16be4412757587057f7cc3446aefd6e6d0cde3cefda"}
Nov 03 20:53:15.821502 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:15.821448    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24" event={"ID":"128dab95-58a5-4fc4-bc64-21b427628fc2","Type":"ContainerStarted","Data":"d3b1ca024d3aebccfa91df1dcccfcffe05f3e2893c4cef0d6f45d4ee3356494f"}
Nov 03 20:53:16.827744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:16.827709    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv" event={"ID":"d8faa8b2-ba59-46b1-8f4c-1a043ac3a595","Type":"ContainerStarted","Data":"8e1e1e41a1eb35a108e4f95a8e696e39137c06bede9623ffebe72f4511dc0e53"}
Nov 03 20:53:16.828150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:16.827793    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:16.959353 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:16.959307    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv" podStartSLOduration=3.042228481 podStartE2EDuration="7.959295714s" totalImagesPullingTime="4.445301415s" totalInitContainerRuntime="471.765818ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:16.958461718 +0000 UTC m=+298.589775191" watchObservedRunningTime="2026-11-03 20:53:16.959295714 +0000 UTC m=+298.590609191"
Nov 03 20:53:18.508059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:18.508029    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:53:18.508506 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:18.508454    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:53:18.519517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:18.519498    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:53:18.519890 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:18.519874    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:53:19.260111 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:53:19.260074    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:53:19.842717 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:19.842683    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24" event={"ID":"128dab95-58a5-4fc4-bc64-21b427628fc2","Type":"ContainerStarted","Data":"2567ab61ac3d5532c2fdcc9f3e264e83a34c2604820fabf99d0721425d16931f"}
Nov 03 20:53:19.843104 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:19.842791    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:19.952464 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:19.952417    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24" podStartSLOduration=2.2973053119999998 podStartE2EDuration="5.952404467s" totalImagesPullingTime="3.655099155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:19.951743329 +0000 UTC m=+301.583056806" watchObservedRunningTime="2026-11-03 20:53:19.952404467 +0000 UTC m=+301.583717944"
Nov 03 20:53:26.638372 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.638333    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 20:53:26.644467 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.644444    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.690573 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.690553    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 20:53:26.714362 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.714336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsn6t\" (UniqueName: \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.714455 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.714397    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.735812 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.735789    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vqxqx\""
Nov 03 20:53:26.814932 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.814910    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsn6t\" (UniqueName: \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.815035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.814971    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.815305 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.815285    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.872508 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.872481    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsn6t\" (UniqueName: \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\") pod \"kuadrant-operator-controller-manager-678d87766f-gn82t\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:26.956737 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:26.956713    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:27.369159 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:27.369134    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 20:53:27.375379 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:27.375358    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:53:27.835457 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:27.835423    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6ffb58b49d-qvqnv"
Nov 03 20:53:27.877067 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:27.877028    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" event={"ID":"e93933ed-7935-46bf-8a67-258fda1d0fd6","Type":"ContainerStarted","Data":"3cde68ced461b9a65333f2eafcf813b52d755f7985c2d0abb9a4701749cfc603"}
Nov 03 20:53:30.849818 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:30.849789    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ngx24"
Nov 03 20:53:32.902727 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:32.902696    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" event={"ID":"e93933ed-7935-46bf-8a67-258fda1d0fd6","Type":"ContainerStarted","Data":"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"}
Nov 03 20:53:32.903139 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:32.902796    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:33.110072 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:33.110022    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" podStartSLOduration=2.466439303 podStartE2EDuration="7.110006418s" totalImagesPullingTime="4.643567115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:33.062572508 +0000 UTC m=+314.693885998" watchObservedRunningTime="2026-11-03 20:53:33.110006418 +0000 UTC m=+314.741319896"
Nov 03 20:53:36.454027 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.453998    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"]
Nov 03 20:53:36.457368 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.457349    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.509370 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.509333    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 20:53:36.510829 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.510807    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"]
Nov 03 20:53:36.553178 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.553155    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 20:53:36.553268 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.553186    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-knms6\""
Nov 03 20:53:36.593424 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.593402    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.593514 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.593472    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.593680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.593631    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4r5\" (UniqueName: \"kubernetes.io/projected/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-kube-api-access-rg4r5\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.694497 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.694474    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4r5\" (UniqueName: \"kubernetes.io/projected/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-kube-api-access-rg4r5\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.694602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.694509    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.694602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.694541    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.695129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.695109    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.696945 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.696922    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.751869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.751816    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4r5\" (UniqueName: \"kubernetes.io/projected/8fac1941-f4d4-4e7d-ad78-a6c93434daf3-kube-api-access-rg4r5\") pod \"kuadrant-console-plugin-66c4fb575d-csxht\" (UID: \"8fac1941-f4d4-4e7d-ad78-a6c93434daf3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:36.768854 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:36.768834    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"
Nov 03 20:53:37.129866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:37.129841    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht"]
Nov 03 20:53:37.132307 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:37.132274    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fac1941_f4d4_4e7d_ad78_a6c93434daf3.slice/crio-4e7e70f1c7598868556e32b8f40f968838146a831a22bbed8a4a12dad8a41243 WatchSource:0}: Error finding container 4e7e70f1c7598868556e32b8f40f968838146a831a22bbed8a4a12dad8a41243: Status 404 returned error can't find the container with id 4e7e70f1c7598868556e32b8f40f968838146a831a22bbed8a4a12dad8a41243
Nov 03 20:53:37.926840 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:37.926807    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht" event={"ID":"8fac1941-f4d4-4e7d-ad78-a6c93434daf3","Type":"ContainerStarted","Data":"4e7e70f1c7598868556e32b8f40f968838146a831a22bbed8a4a12dad8a41243"}
Nov 03 20:53:42.951495 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:42.951460    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht" event={"ID":"8fac1941-f4d4-4e7d-ad78-a6c93434daf3","Type":"ContainerStarted","Data":"3d0be256ae18ce3578f80f6378cc083c71a35726d26d3c1dcc3f51eb28ac259f"}
Nov 03 20:53:43.061556 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.061472    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-csxht" podStartSLOduration=1.4415269880000001 podStartE2EDuration="7.06145959s" totalImagesPullingTime="5.619932602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:43.060982123 +0000 UTC m=+324.692295600" watchObservedRunningTime="2026-11-03 20:53:43.06145959 +0000 UTC m=+324.692773066"
Nov 03 20:53:43.396517 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.396446    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"]
Nov 03 20:53:43.403301 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.403280    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.452698 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.452668    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vm4jx\""
Nov 03 20:53:43.458917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.458890    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"]
Nov 03 20:53:43.548212 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548183    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548215    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548331 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548247    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548420 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548372    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548420 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548404    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548483 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548430    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548532 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548518    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08564a25-5b1d-419c-b108-18b2c11d7f32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548568 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548547    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.548568 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.548564    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cn7\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-kube-api-access-x5cn7\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649623 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649569    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649623 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649598    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649623 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649617    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649828 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649654    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08564a25-5b1d-419c-b108-18b2c11d7f32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649828 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649828 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649723    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cn7\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-kube-api-access-x5cn7\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649969 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649911    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.649969 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.649962    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650083 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650003    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650101    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650200 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650181    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650272 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650256    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650413 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650390    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.650549 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.650527    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08564a25-5b1d-419c-b108-18b2c11d7f32-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.652240 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.652217    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.652541 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.652522    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.707160 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.707130    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.707299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.707280    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cn7\" (UniqueName: \"kubernetes.io/projected/08564a25-5b1d-419c-b108-18b2c11d7f32-kube-api-access-x5cn7\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc\" (UID: \"08564a25-5b1d-419c-b108-18b2c11d7f32\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.718092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.718073    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:43.909386 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:43.909316    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 20:53:44.179261 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.179235    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"]
Nov 03 20:53:44.181537 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:44.181501    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08564a25_5b1d_419c_b108_18b2c11d7f32.slice/crio-57534d0997d1b86d1a99b16dd0d9f1f39291b75e752ed3dcb5b7fbcb9eff300e WatchSource:0}: Error finding container 57534d0997d1b86d1a99b16dd0d9f1f39291b75e752ed3dcb5b7fbcb9eff300e: Status 404 returned error can't find the container with id 57534d0997d1b86d1a99b16dd0d9f1f39291b75e752ed3dcb5b7fbcb9eff300e
Nov 03 20:53:44.183648 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.183607    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.183735 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.183688    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.183735 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.183721    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.962312 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.962271    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc" event={"ID":"08564a25-5b1d-419c-b108-18b2c11d7f32","Type":"ContainerStarted","Data":"fe2b7eb9fea03768349deedac85ee15918fde9d36c4c1a86684aad12f6f1f659"}
Nov 03 20:53:44.962479 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:44.962341    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc" event={"ID":"08564a25-5b1d-419c-b108-18b2c11d7f32","Type":"ContainerStarted","Data":"57534d0997d1b86d1a99b16dd0d9f1f39291b75e752ed3dcb5b7fbcb9eff300e"}
Nov 03 20:53:45.079006 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.078961    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc" podStartSLOduration=2.078947296 podStartE2EDuration="2.078947296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:45.077997151 +0000 UTC m=+326.709310628" watchObservedRunningTime="2026-11-03 20:53:45.078947296 +0000 UTC m=+326.710260774"
Nov 03 20:53:45.698958 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.698918    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:53:45.702665 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.702628    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.718530 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.718507    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:45.723119 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.723097    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:45.755304 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.755279    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 20:53:45.757354 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.757330    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:53:45.864196 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.864159    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7lr\" (UniqueName: \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.864338 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.864217    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.947757 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.947733    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:53:45.964692 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.964608    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7lr\" (UniqueName: \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.964692 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.964665    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.965406 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.965385    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:45.967953 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.967938    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:45.968730 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:45.968715    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc"
Nov 03 20:53:46.025610 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:46.025579    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7lr\" (UniqueName: \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\") pod \"limitador-limitador-74dfcfb894-ggxxr\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:46.319441 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:46.319372    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:46.680007 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:46.679987    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:53:46.682451 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:46.682425    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096b8ed9_f419_4114_b181_e0f062d76ce6.slice/crio-932755bdfab27000c3141a1311053bb3f020adfba2d6d40d7c4286ccc81d5027 WatchSource:0}: Error finding container 932755bdfab27000c3141a1311053bb3f020adfba2d6d40d7c4286ccc81d5027: Status 404 returned error can't find the container with id 932755bdfab27000c3141a1311053bb3f020adfba2d6d40d7c4286ccc81d5027
Nov 03 20:53:46.974281 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:46.974202    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" event={"ID":"096b8ed9-f419-4114-b181-e0f062d76ce6","Type":"ContainerStarted","Data":"932755bdfab27000c3141a1311053bb3f020adfba2d6d40d7c4286ccc81d5027"}
Nov 03 20:53:48.794820 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.794794    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:53:48.797917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.797902    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:48.890734 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.890649    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-config-file\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:48.890734 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.890692    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6428\" (UniqueName: \"kubernetes.io/projected/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-kube-api-access-c6428\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:48.895738 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.895716    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:53:48.983152 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.983128    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" event={"ID":"096b8ed9-f419-4114-b181-e0f062d76ce6","Type":"ContainerStarted","Data":"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"}
Nov 03 20:53:48.983271 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.983253    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:53:48.991675 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.991653    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-config-file\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:48.991747 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.991688    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6428\" (UniqueName: \"kubernetes.io/projected/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-kube-api-access-c6428\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:48.992259 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:48.992242    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-config-file\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:49.049783 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.049760    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6428\" (UniqueName: \"kubernetes.io/projected/18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3-kube-api-access-c6428\") pod \"limitador-limitador-7c9b87d574-kcbgn\" (UID: \"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:49.093357 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.093311    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" podStartSLOduration=2.140474611 podStartE2EDuration="4.093299589s" totalImagesPullingTime="1.952824978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:49.092514291 +0000 UTC m=+330.723827769" watchObservedRunningTime="2026-11-03 20:53:49.093299589 +0000 UTC m=+330.724613065"
Nov 03 20:53:49.109065 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.109047    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:49.126783 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.126758    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:53:49.259499 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:53:49.259468    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:53:49.478821 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.478764    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:53:49.486695 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:53:49.486667    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18eb3d6d_1d38_4c34_a7f2_46092ddb2ad3.slice/crio-c49e255adbb333c41564f65d84df9071c43ff6353f58fb314df348d8279c8b63 WatchSource:0}: Error finding container c49e255adbb333c41564f65d84df9071c43ff6353f58fb314df348d8279c8b63: Status 404 returned error can't find the container with id c49e255adbb333c41564f65d84df9071c43ff6353f58fb314df348d8279c8b63
Nov 03 20:53:49.989097 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.989062    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn" event={"ID":"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3","Type":"ContainerStarted","Data":"501cbdda0c25b63f546c35cacefab64df0e879ec4702de87520dfa78d233e73c"}
Nov 03 20:53:49.989586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.989123    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn" event={"ID":"18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3","Type":"ContainerStarted","Data":"c49e255adbb333c41564f65d84df9071c43ff6353f58fb314df348d8279c8b63"}
Nov 03 20:53:49.989586 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:49.989269    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:53:50.101981 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:50.101936    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn" podStartSLOduration=2.101923762 podStartE2EDuration="2.101923762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:50.100806069 +0000 UTC m=+331.732119547" watchObservedRunningTime="2026-11-03 20:53:50.101923762 +0000 UTC m=+331.733237239"
Nov 03 20:53:59.990575 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:53:59.990537    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:54:00.994880 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:00.994852    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"
Nov 03 20:54:01.152192 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.152163    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:54:01.152395 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.152356    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" podUID="096b8ed9-f419-4114-b181-e0f062d76ce6" containerName="limitador" containerID="cri-o://f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641" gracePeriod=30
Nov 03 20:54:01.680879 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.680856    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:54:01.683882 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.683864    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\" (UniqueName: \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\") pod \"096b8ed9-f419-4114-b181-e0f062d76ce6\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") "
Nov 03 20:54:01.683961 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.683897    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\" (UniqueName: \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\") pod \"096b8ed9-f419-4114-b181-e0f062d76ce6\" (UID: \"096b8ed9-f419-4114-b181-e0f062d76ce6\") "
Nov 03 20:54:01.684266 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.684243    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file" pod "096b8ed9-f419-4114-b181-e0f062d76ce6" (UID: "096b8ed9-f419-4114-b181-e0f062d76ce6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:01.685890 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.685867    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr" pod "096b8ed9-f419-4114-b181-e0f062d76ce6" (UID: "096b8ed9-f419-4114-b181-e0f062d76ce6"). InnerVolumeSpecName "kube-api-access-wx7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:01.784998 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.784932    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7lr\" (UniqueName: \"kubernetes.io/projected/096b8ed9-f419-4114-b181-e0f062d76ce6-kube-api-access-wx7lr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:01.784998 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:01.784957    5533 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/096b8ed9-f419-4114-b181-e0f062d76ce6-config-file\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:02.043897 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.043825    5533 generic.go:356] "Generic (PLEG): container finished" podID="096b8ed9-f419-4114-b181-e0f062d76ce6" containerID="f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641" exitCode=0
Nov 03 20:54:02.043897 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.043870    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"
Nov 03 20:54:02.044336 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.043917    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" event={"ID":"096b8ed9-f419-4114-b181-e0f062d76ce6","Type":"ContainerDied","Data":"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"}
Nov 03 20:54:02.044336 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.043996    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ggxxr" event={"ID":"096b8ed9-f419-4114-b181-e0f062d76ce6","Type":"ContainerDied","Data":"932755bdfab27000c3141a1311053bb3f020adfba2d6d40d7c4286ccc81d5027"}
Nov 03 20:54:02.044336 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.044020    5533 scope.go:122] "RemoveContainer" containerID="f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"
Nov 03 20:54:02.053081 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.053065    5533 scope.go:122] "RemoveContainer" containerID="f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"
Nov 03 20:54:02.053340 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:54:02.053318    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641\": container with ID starting with f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641 not found: ID does not exist" containerID="f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"
Nov 03 20:54:02.053404 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.053346    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641"} err="failed to get container status \"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641\": rpc error: code = NotFound desc = could not find container \"f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641\": container with ID starting with f7ccd2777ca918a8a1b114c4bdfc2255d0ec41eb0a0bf952d8117949d7382641 not found: ID does not exist"
Nov 03 20:54:02.198139 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.198115    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:54:02.244921 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.244901    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ggxxr"]
Nov 03 20:54:02.490971 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:02.490945    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096b8ed9-f419-4114-b181-e0f062d76ce6" path="/var/lib/kubelet/pods/096b8ed9-f419-4114-b181-e0f062d76ce6/volumes"
Nov 03 20:54:09.021658 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.021612    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"]
Nov 03 20:54:09.022016 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.021943    5533 cpu_manager.go:414] "containerMap: removing container" podUID="096b8ed9-f419-4114-b181-e0f062d76ce6" containerName="limitador"
Nov 03 20:54:09.022016 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.021956    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096b8ed9-f419-4114-b181-e0f062d76ce6" containerName="limitador"
Nov 03 20:54:09.022096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.022019    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="096b8ed9-f419-4114-b181-e0f062d76ce6" containerName="limitador"
Nov 03 20:54:09.024585 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.024564    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.029989 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.029963    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078be369-f4bf-4802-ab25-05a5c196b0d7-data\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.030113 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.030030    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c655q\" (UniqueName: \"kubernetes.io/projected/078be369-f4bf-4802-ab25-05a5c196b0d7-kube-api-access-c655q\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.120298 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.120268    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:09.120298 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.120273    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 20:54:09.120298 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.120289    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-msh2n\""
Nov 03 20:54:09.120298 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.120300    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:09.122157 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.122139    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"]
Nov 03 20:54:09.131054 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.131036    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c655q\" (UniqueName: \"kubernetes.io/projected/078be369-f4bf-4802-ab25-05a5c196b0d7-kube-api-access-c655q\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.131141 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.131075    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078be369-f4bf-4802-ab25-05a5c196b0d7-data\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.131688 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.131666    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078be369-f4bf-4802-ab25-05a5c196b0d7-data\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.188473 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.188445    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c655q\" (UniqueName: \"kubernetes.io/projected/078be369-f4bf-4802-ab25-05a5c196b0d7-kube-api-access-c655q\") pod \"postgres-7cdbf6dc44-6ndhn\" (UID: \"078be369-f4bf-4802-ab25-05a5c196b0d7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.336209 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.336156    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:09.708749 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:09.708724    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"]
Nov 03 20:54:09.710818 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:54:09.710783    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078be369_f4bf_4802_ab25_05a5c196b0d7.slice/crio-43d443aa4dda42955e4dda0cebaa4e9293941300bdc40d42eb73f7019be91333 WatchSource:0}: Error finding container 43d443aa4dda42955e4dda0cebaa4e9293941300bdc40d42eb73f7019be91333: Status 404 returned error can't find the container with id 43d443aa4dda42955e4dda0cebaa4e9293941300bdc40d42eb73f7019be91333
Nov 03 20:54:10.083318 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:10.083251    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn" event={"ID":"078be369-f4bf-4802-ab25-05a5c196b0d7","Type":"ContainerStarted","Data":"43d443aa4dda42955e4dda0cebaa4e9293941300bdc40d42eb73f7019be91333"}
Nov 03 20:54:19.125930 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:19.125888    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn" event={"ID":"078be369-f4bf-4802-ab25-05a5c196b0d7","Type":"ContainerStarted","Data":"faebf1bc48068d75f3494134ced355014f18542f1e419cbe4624c1bfd9b7bef0"}
Nov 03 20:54:19.126347 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:19.125976    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:19.235421 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:19.235371    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn" podStartSLOduration=2.747563957 podStartE2EDuration="11.235357052s" totalImagesPullingTime="8.487793095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:19.233777143 +0000 UTC m=+360.865090619" watchObservedRunningTime="2026-11-03 20:54:19.235357052 +0000 UTC m=+360.866670528"
Nov 03 20:54:19.267145 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:54:19.262037    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:54:23.995270 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:23.995232    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"]
Nov 03 20:54:23.998163 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:23.998143    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.047177 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.047150    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bcf5edfd-efee-482e-be0c-3cca269363ca-manager-config\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.047299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.047195    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-metrics-certs\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.047299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.047272    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-cert\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.047379 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.047302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz425\" (UniqueName: \"kubernetes.io/projected/bcf5edfd-efee-482e-be0c-3cca269363ca-kube-api-access-jz425\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.094442 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.094417    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 20:54:24.094442 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.094438    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 20:54:24.094442 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.094424    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-gcrdg\""
Nov 03 20:54:24.094707 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.094541    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 20:54:24.096289 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.096266    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"]
Nov 03 20:54:24.147707 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.147686    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-metrics-certs\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.147812 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.147734    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-cert\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.147812 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.147761    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz425\" (UniqueName: \"kubernetes.io/projected/bcf5edfd-efee-482e-be0c-3cca269363ca-kube-api-access-jz425\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.147897 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.147825    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bcf5edfd-efee-482e-be0c-3cca269363ca-manager-config\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.148467 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.148445    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bcf5edfd-efee-482e-be0c-3cca269363ca-manager-config\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.150074 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.150054    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-cert\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.150151 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.150127    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf5edfd-efee-482e-be0c-3cca269363ca-metrics-certs\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.205347 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.205325    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz425\" (UniqueName: \"kubernetes.io/projected/bcf5edfd-efee-482e-be0c-3cca269363ca-kube-api-access-jz425\") pod \"jobset-controller-manager-7c74f567cc-mkn89\" (UID: \"bcf5edfd-efee-482e-be0c-3cca269363ca\") " pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.310200 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.310150    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:24.671445 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:24.671419    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"]
Nov 03 20:54:24.673895 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:54:24.673866    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf5edfd_efee_482e_be0c_3cca269363ca.slice/crio-80157cb92af222804029094ce6e98bb7a23c2c877a468f71b7ec5947d3c9256b WatchSource:0}: Error finding container 80157cb92af222804029094ce6e98bb7a23c2c877a468f71b7ec5947d3c9256b: Status 404 returned error can't find the container with id 80157cb92af222804029094ce6e98bb7a23c2c877a468f71b7ec5947d3c9256b
Nov 03 20:54:25.153660 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:25.153560    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89" event={"ID":"bcf5edfd-efee-482e-be0c-3cca269363ca","Type":"ContainerStarted","Data":"80157cb92af222804029094ce6e98bb7a23c2c877a468f71b7ec5947d3c9256b"}
Nov 03 20:54:25.159820 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:25.159799    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6ndhn"
Nov 03 20:54:41.226441 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:41.226387    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89" event={"ID":"bcf5edfd-efee-482e-be0c-3cca269363ca","Type":"ContainerStarted","Data":"d3a046d956e005165ba8a280bc01be1ffe44f5ae2ccbda05a7399769c273ae8d"}
Nov 03 20:54:41.226838 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:41.226476    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:54:41.336742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:41.336693    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89" podStartSLOduration=2.288537918 podStartE2EDuration="18.336678903s" totalImagesPullingTime="16.048140985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:41.336013478 +0000 UTC m=+382.967326955" watchObservedRunningTime="2026-11-03 20:54:41.336678903 +0000 UTC m=+382.967992380"
Nov 03 20:54:49.259588 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:54:49.259543    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:54:52.236070 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:54:52.236035    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7c74f567cc-mkn89"
Nov 03 20:55:19.259808 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:55:19.259774    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:55:21.633748 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:21.633715    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:55:30.459901 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.459872    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerID="1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e" exitCode=0
Nov 03 20:55:30.460276 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.459946    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerDied","Data":"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"}
Nov 03 20:55:30.460319 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.460281    5533 scope.go:122] "RemoveContainer" containerID="1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"
Nov 03 20:55:30.461344 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.461318    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:30.461407 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.461385    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:30.461444 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:30.461417    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:55:31.479104 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:31.479069    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerStarted","Data":"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"}
Nov 03 20:55:31.593988 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:31.593954    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:55:31.594152 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:31.594117    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Nov 03 20:55:31.594195 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:31.594166    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Nov 03 20:55:32.594361 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:32.594331    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Nov 03 20:55:32.594737 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:32.594380    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Nov 03 20:55:33.594067 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:33.594032    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Nov 03 20:55:33.594246 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:33.594093    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Nov 03 20:55:34.594015 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:34.593981    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Nov 03 20:55:34.594395 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:34.594044    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Nov 03 20:55:35.594106 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:35.594072    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Nov 03 20:55:35.594500 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:35.594122    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Nov 03 20:55:36.597727 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:36.597698    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:55:36.598100 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:36.597963    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:55:36.598678 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:55:36.598658    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 20:55:49.260847 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:55:49.260808    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:56:19.261244 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:56:19.261156    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:56:49.261108 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:56:49.261068    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:57:19.260440 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:57:19.260405    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:57:36.601379 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:36.601341    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:57:39.357766 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.357733    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"]
Nov 03 20:57:39.361010 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.360988    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375363 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375335    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375461 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375367    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375461 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375389    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375537 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375504    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375570 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375555    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wvz\" (UniqueName: \"kubernetes.io/projected/a9a714ca-1b70-4236-ab4f-022c4c3449ef-kube-api-access-47wvz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.375606 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.375590    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.457166 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.457135    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:57:39.457166 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.457148    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gh4x2\""
Nov 03 20:57:39.457166 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.457141    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 20:57:39.457166 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.457143    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:57:39.458721 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.458703    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"]
Nov 03 20:57:39.476671 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476627    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.476770 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476678    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.476770 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476713    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.476770 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476743    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.476918 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476781    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wvz\" (UniqueName: \"kubernetes.io/projected/a9a714ca-1b70-4236-ab4f-022c4c3449ef-kube-api-access-47wvz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.476918 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.476829    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.477158 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.477139    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.477158 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.477147    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.477261 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.477174    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.479008 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.478991    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a714ca-1b70-4236-ab4f-022c4c3449ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.479296 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.479270    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a714ca-1b70-4236-ab4f-022c4c3449ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.534779 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.534749    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wvz\" (UniqueName: \"kubernetes.io/projected/a9a714ca-1b70-4236-ab4f-022c4c3449ef-kube-api-access-47wvz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-89cxr\" (UID: \"a9a714ca-1b70-4236-ab4f-022c4c3449ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:39.673177 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:39.673156    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:40.036952 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:40.036897    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"]
Nov 03 20:57:40.039171 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:57:40.039142    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a714ca_1b70_4236_ab4f_022c4c3449ef.slice/crio-faca958fd75e3f1a31d7660add8833b08c7eb0049a26edfa55c7b2101a8319f2 WatchSource:0}: Error finding container faca958fd75e3f1a31d7660add8833b08c7eb0049a26edfa55c7b2101a8319f2: Status 404 returned error can't find the container with id faca958fd75e3f1a31d7660add8833b08c7eb0049a26edfa55c7b2101a8319f2
Nov 03 20:57:40.058074 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:40.058044    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr" event={"ID":"a9a714ca-1b70-4236-ab4f-022c4c3449ef","Type":"ContainerStarted","Data":"faca958fd75e3f1a31d7660add8833b08c7eb0049a26edfa55c7b2101a8319f2"}
Nov 03 20:57:42.069262 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:42.069226    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr" event={"ID":"a9a714ca-1b70-4236-ab4f-022c4c3449ef","Type":"ContainerStarted","Data":"e47a9bf5ea73cc4f2ba2aae137866cb6ab85e6de4f28177058d181e19ae9909f"}
Nov 03 20:57:42.181049 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:42.181005    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr" podStartSLOduration=1.656757056 podStartE2EDuration="3.180991578s" totalImagesPullingTime="1.524234522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:42.180019538 +0000 UTC m=+563.811333016" watchObservedRunningTime="2026-11-03 20:57:42.180991578 +0000 UTC m=+563.812305056"
Nov 03 20:57:42.674165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:42.674127    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:46.851161 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.851127    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"]
Nov 03 20:57:46.854495 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.854476    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933428    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933460    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk569\" (UniqueName: \"kubernetes.io/projected/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-kube-api-access-gk569\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933483    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933559    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933601    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.933712 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.933666    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:46.950323 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.950301    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 20:57:46.952016 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:46.951998    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"]
Nov 03 20:57:47.034624 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034594    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.034794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034654    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.034794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034689    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.034794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034709    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.034794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034777    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.035009 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.034806    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk569\" (UniqueName: \"kubernetes.io/projected/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-kube-api-access-gk569\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.035215 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.035182    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.035322 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.035219    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.035322 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.035239    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.037009 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.036983    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.037250 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.037235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.093328 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.093298    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk569\" (UniqueName: \"kubernetes.io/projected/7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21-kube-api-access-gk569\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-5vdsp\" (UID: \"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.166789 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.166729    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:47.529615 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:47.529443    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"]
Nov 03 20:57:47.532399 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:57:47.532366    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdac0ca_f0ca_4565_93cb_b8b8a42b6c21.slice/crio-2879f3b566eee8e9861141ae2e6ca4868d579106199a19522e4458c03059c428 WatchSource:0}: Error finding container 2879f3b566eee8e9861141ae2e6ca4868d579106199a19522e4458c03059c428: Status 404 returned error can't find the container with id 2879f3b566eee8e9861141ae2e6ca4868d579106199a19522e4458c03059c428
Nov 03 20:57:48.099848 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:48.099808    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp" event={"ID":"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21","Type":"ContainerStarted","Data":"2879f3b566eee8e9861141ae2e6ca4868d579106199a19522e4458c03059c428"}
Nov 03 20:57:49.105882 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.105844    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp" event={"ID":"7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21","Type":"ContainerStarted","Data":"683c98548c9945c46cbaa73b99269dd49deb1ea0105269cf192f23c27e8882bc"}
Nov 03 20:57:49.166813 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.166774    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:49.218606 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.218557    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp" podStartSLOduration=2.606804775 podStartE2EDuration="3.218544052s" totalImagesPullingTime="611.739277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:49.217765598 +0000 UTC m=+570.849079074" watchObservedRunningTime="2026-11-03 20:57:49.218544052 +0000 UTC m=+570.849857508"
Nov 03 20:57:49.258566 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:57:49.258536    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:57:49.265835 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.265812    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:57:49.673417 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.673386    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:49.686080 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:49.686055    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:50.125697 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:50.125604    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-89cxr"
Nov 03 20:57:53.747142 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.747108    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vfz7m"]
Nov 03 20:57:53.750654 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.750615    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.789842 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.789812    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.789975 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.789847    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d34006-15de-452d-a377-c7bd2f14bdeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.789975 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.789863    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqqz\" (UniqueName: \"kubernetes.io/projected/b3d34006-15de-452d-a377-c7bd2f14bdeb-kube-api-access-ppqqz\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.789975 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.789889    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.790135 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.790026    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.790135 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.790086    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.799967 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.799947    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 20:57:53.848027 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.848003    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vfz7m"]
Nov 03 20:57:53.890787 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890755    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.890917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890791    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d34006-15de-452d-a377-c7bd2f14bdeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.890917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890815    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqqz\" (UniqueName: \"kubernetes.io/projected/b3d34006-15de-452d-a377-c7bd2f14bdeb-kube-api-access-ppqqz\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.890917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890850    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.890917 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890880    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.891096 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.890931    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.891204 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.891183    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.891289 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.891257    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.891351 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.891270    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-home\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.893091 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.893076    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d34006-15de-452d-a377-c7bd2f14bdeb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.893385 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.893369    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d34006-15de-452d-a377-c7bd2f14bdeb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:53.948188 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:53.948159    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqqz\" (UniqueName: \"kubernetes.io/projected/b3d34006-15de-452d-a377-c7bd2f14bdeb-kube-api-access-ppqqz\") pod \"echo-simulator-kserve-64b6f8f96c-vfz7m\" (UID: \"b3d34006-15de-452d-a377-c7bd2f14bdeb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:54.062889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:54.062832    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:54.429673 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:54.429630    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vfz7m"]
Nov 03 20:57:54.431573 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:57:54.431541    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d34006_15de_452d_a377_c7bd2f14bdeb.slice/crio-364942f0f395c464acc0e5c036dc6bf0de3c419835fd4606a7299d50b7e31830 WatchSource:0}: Error finding container 364942f0f395c464acc0e5c036dc6bf0de3c419835fd4606a7299d50b7e31830: Status 404 returned error can't find the container with id 364942f0f395c464acc0e5c036dc6bf0de3c419835fd4606a7299d50b7e31830
Nov 03 20:57:55.143594 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:55.143558    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m" event={"ID":"b3d34006-15de-452d-a377-c7bd2f14bdeb","Type":"ContainerStarted","Data":"364942f0f395c464acc0e5c036dc6bf0de3c419835fd4606a7299d50b7e31830"}
Nov 03 20:57:57.153789 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:57.153753    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m" event={"ID":"b3d34006-15de-452d-a377-c7bd2f14bdeb","Type":"ContainerStarted","Data":"a7fbd227ee184149d871733737d2feadca0b4f8acde1f4c2e292dc67f953f0c3"}
Nov 03 20:57:57.167186 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:57.167163    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:57.179952 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:57.179930    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:57:57.266589 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:57.266510    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m" podStartSLOduration=1.713574478 podStartE2EDuration="4.266497768s" totalImagesPullingTime="2.55292329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:57.265715365 +0000 UTC m=+578.897028898" watchObservedRunningTime="2026-11-03 20:57:57.266497768 +0000 UTC m=+578.897811244"
Nov 03 20:57:58.063356 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:58.063294    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:57:58.177510 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:57:58.177481    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-5vdsp"
Nov 03 20:58:03.682882 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.682841    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"]
Nov 03 20:58:03.686769 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.686745    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769059 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769031    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4da839-0fd8-4d9e-81a2-371496377bba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769175 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769088    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vdq\" (UniqueName: \"kubernetes.io/projected/3b4da839-0fd8-4d9e-81a2-371496377bba-kube-api-access-j7vdq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769220 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769192    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769255 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769226    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769255 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769243    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.769352 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.769257    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.782274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.782250    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 20:58:03.784013 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.783992    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"]
Nov 03 20:58:03.870294 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870262    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vdq\" (UniqueName: \"kubernetes.io/projected/3b4da839-0fd8-4d9e-81a2-371496377bba-kube-api-access-j7vdq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870319    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870432    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870473    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870516    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870761 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870572    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4da839-0fd8-4d9e-81a2-371496377bba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870844    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870930 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870871    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.870982 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.870951    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.872501 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.872481    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b4da839-0fd8-4d9e-81a2-371496377bba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.872824 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.872805    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4da839-0fd8-4d9e-81a2-371496377bba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.928245 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.928216    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vdq\" (UniqueName: \"kubernetes.io/projected/3b4da839-0fd8-4d9e-81a2-371496377bba-kube-api-access-j7vdq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq\" (UID: \"3b4da839-0fd8-4d9e-81a2-371496377bba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:03.998327 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:03.998274    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:04.064250 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:04.064217    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:58:04.078011 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:04.077983    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:58:04.199441 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:04.199415    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vfz7m"
Nov 03 20:58:04.360094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:04.360073    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"]
Nov 03 20:58:04.362672 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:58:04.362621    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4da839_0fd8_4d9e_81a2_371496377bba.slice/crio-95c3c404030a71561a1e440f536bb9557e6eae6fcbbffc4e9e458b805dfb7fc4 WatchSource:0}: Error finding container 95c3c404030a71561a1e440f536bb9557e6eae6fcbbffc4e9e458b805dfb7fc4: Status 404 returned error can't find the container with id 95c3c404030a71561a1e440f536bb9557e6eae6fcbbffc4e9e458b805dfb7fc4
Nov 03 20:58:05.194120 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:05.194087    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" event={"ID":"3b4da839-0fd8-4d9e-81a2-371496377bba","Type":"ContainerStarted","Data":"aad1c8dc26b33bc4fb95fd3059acc9feebd99e0e57fee4442ba0d5de15a9f622"}
Nov 03 20:58:05.194507 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:05.194131    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" event={"ID":"3b4da839-0fd8-4d9e-81a2-371496377bba","Type":"ContainerStarted","Data":"95c3c404030a71561a1e440f536bb9557e6eae6fcbbffc4e9e458b805dfb7fc4"}
Nov 03 20:58:05.304574 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:05.304489    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podStartSLOduration=1.675498782 podStartE2EDuration="2.30447697s" totalImagesPullingTime="628.978188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:05.30376103 +0000 UTC m=+586.935074507" watchObservedRunningTime="2026-11-03 20:58:05.30447697 +0000 UTC m=+586.935790483"
Nov 03 20:58:05.998393 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:05.998363    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:06.375362 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:06.375287    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:58:13.998440 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:13.998411    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:14.010722 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:14.010701    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:14.247319 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:14.247292    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 20:58:18.543437 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:18.543408    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:58:18.543882 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:18.543408    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 20:58:18.555060 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:18.555041    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:58:18.555184 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:18.555061    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 20:58:19.261355 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:58:19.261324    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:58:26.152613 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.152579    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 20:58:26.157351 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.157329    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250273 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250242    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250415 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250294    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250415 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250317    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250536 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250536 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250442    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250536 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250471    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250582    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250608    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.250726 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.250623    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtztm\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.255368 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.255344    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 20:58:26.351015 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.350988    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351025    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351040    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtztm\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351060    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351079    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351096    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351137    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351166    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351200    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351747 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351718    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351759    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.351866 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351843    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.352012 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.351960    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.352097 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.352074    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.353743 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.353721    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.353839 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.353788    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.408552 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.408500    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.408681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.408662    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtztm\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-f7mxx\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.469916 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.469898    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:26.883744 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.883717    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 20:58:26.885853 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:58:26.885826    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c10d5f_84ff_44c1_9364_372f9358ff91.slice/crio-b25da6bf38e920b41347f89ab82314c0f73317bc2053ab9e3866d4cb36e5b692 WatchSource:0}: Error finding container b25da6bf38e920b41347f89ab82314c0f73317bc2053ab9e3866d4cb36e5b692: Status 404 returned error can't find the container with id b25da6bf38e920b41347f89ab82314c0f73317bc2053ab9e3866d4cb36e5b692
Nov 03 20:58:26.887870 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.887839    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:26.887936 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.887904    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:26.887977 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:26.887936    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:58:27.305680 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.305644    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" event={"ID":"76c10d5f-84ff-44c1-9364-372f9358ff91","Type":"ContainerStarted","Data":"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"}
Nov 03 20:58:27.306130 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.305697    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" event={"ID":"76c10d5f-84ff-44c1-9364-372f9358ff91","Type":"ContainerStarted","Data":"b25da6bf38e920b41347f89ab82314c0f73317bc2053ab9e3866d4cb36e5b692"}
Nov 03 20:58:27.428478 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.428425    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podStartSLOduration=1.42840907 podStartE2EDuration="1.42840907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:27.426765565 +0000 UTC m=+609.058079044" watchObservedRunningTime="2026-11-03 20:58:27.42840907 +0000 UTC m=+609.059722547"
Nov 03 20:58:27.475810 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.475770    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:27.477276 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.477249    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:27.477414 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:27.477304    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:28.471609 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:28.471576    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:28.472198 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:28.472167    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:29.471024 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:29.470988    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:29.471332 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:29.471303    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:30.471038 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:30.470995    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:30.471681 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:30.471621    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:31.473033 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:31.472997    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:31.473623 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:31.473588    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:32.471973 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:32.471937    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:32.472244 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:32.472214    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:33.470859 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:33.470829    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:58:33.471225 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:33.470880    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 20:58:35.168168 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:35.168133    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:35.168743 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:35.168720    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:35.169421 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:58:35.169399    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 20:58:49.260407 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:58:49.260371    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:59:05.403272 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.403185    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 20:59:05.406834 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.406804    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.456788 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.456763    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-367d7739-gw-openshift-default-dockercfg-hg6zz\""
Nov 03 20:59:05.465101 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465074    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465225 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465108    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465225 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465132    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465225 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465160    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465370 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465251    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465416 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465399    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465455 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465431    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxksj\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465564 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465547    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.465618 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.465581    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.505035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.505008    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 20:59:05.566755 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566732    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.566892 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566759    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.566892 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566782    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.566892 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.566892 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566830    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.566892 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566874    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567112 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566899    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxksj\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567112 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567112 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.566993    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567270 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.567173    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567474 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.567431    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567582 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.567553    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567719 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.567697    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.567816 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.567793    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.569144 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.569123    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.569688 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.569669    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.625792 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.625757    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.627148 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.627124    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxksj\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\") pod \"e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") " pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:05.719460 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:05.719438    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:06.142765 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.142687    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 20:59:06.177557 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:06.177524    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89c6e7a_3aec_4b48_8467_bb27be1240c9.slice/crio-f7afbd6337cd660653a6a6b0a539fc92ce96176e868a4c6ea72af8299166dc1d WatchSource:0}: Error finding container f7afbd6337cd660653a6a6b0a539fc92ce96176e868a4c6ea72af8299166dc1d: Status 404 returned error can't find the container with id f7afbd6337cd660653a6a6b0a539fc92ce96176e868a4c6ea72af8299166dc1d
Nov 03 20:59:06.179624 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.179592    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:06.179768 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.179692    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:06.179768 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.179739    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:06.497118 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.497084    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerStarted","Data":"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"}
Nov 03 20:59:06.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.497151    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerStarted","Data":"f7afbd6337cd660653a6a6b0a539fc92ce96176e868a4c6ea72af8299166dc1d"}
Nov 03 20:59:06.615063 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.614984    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podStartSLOduration=1.614964895 podStartE2EDuration="1.614964895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:06.612121877 +0000 UTC m=+648.243435359" watchObservedRunningTime="2026-11-03 20:59:06.614964895 +0000 UTC m=+648.246278385"
Nov 03 20:59:06.719948 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.719916    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:06.967245 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.967213    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 20:59:06.970696 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:06.970663    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.020094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.020067    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b32d5a5-openshift-default-dockercfg-rqvrz\""
Nov 03 20:59:07.030277 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.030252    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 20:59:07.078169 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078131    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078318 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078188    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078318 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078257    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078408 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078314    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh84c\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078408 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078388    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078503 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078407    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078503 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078443    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078583 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078516    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.078583 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.078575    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179002 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.178960    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179002 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179012    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179038    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179069    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179091    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179114    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179138    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh84c\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179193    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179251 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179218    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179610 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179610 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179509    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179801 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179776    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.179871 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.179801    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.180287 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.180261    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.181499 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.181475    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.181783 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.181763    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.240019 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.239951    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.240188 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.240168    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh84c\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\") pod \"e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") " pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.291017 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.290992    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:07.711627 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.711600    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 20:59:07.713460 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:07.713435    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d753cb_d856_46ee_a29d_a1b4fd36adf6.slice/crio-4bc67c2e954d1b990619d4b08e268d30316f355de666d424eaf3f3a482346f99 WatchSource:0}: Error finding container 4bc67c2e954d1b990619d4b08e268d30316f355de666d424eaf3f3a482346f99: Status 404 returned error can't find the container with id 4bc67c2e954d1b990619d4b08e268d30316f355de666d424eaf3f3a482346f99
Nov 03 20:59:07.716137 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.716105    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:07.716221 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.716165    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:07.716221 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.716197    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:07.725206 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:07.725187    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:08.508121 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:08.508080    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerStarted","Data":"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"}
Nov 03 20:59:08.508299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:08.508142    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerStarted","Data":"4bc67c2e954d1b990619d4b08e268d30316f355de666d424eaf3f3a482346f99"}
Nov 03 20:59:08.508299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:08.508285    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:08.509269 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:08.509248    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:08.621889 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:08.621844    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podStartSLOduration=2.621829971 podStartE2EDuration="2.621829971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:08.620564868 +0000 UTC m=+650.251878346" watchObservedRunningTime="2026-11-03 20:59:08.621829971 +0000 UTC m=+650.253143449"
Nov 03 20:59:09.291772 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:09.291742    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:09.296886 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:09.296861    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:09.513459 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:09.513427    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:09.514210 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:09.514189    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:16.134602 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.134564    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 20:59:16.138376 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.138357    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.187574 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.187554    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a30f61a\""
Nov 03 20:59:16.234420 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.234395    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a30f61a-dockercfg-x69v2\""
Nov 03 20:59:16.237444 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.237419    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 20:59:16.266044 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.265984    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.266428 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.266061    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnw5p\" (UniqueName: \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.367128 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.367096    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnw5p\" (UniqueName: \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.367259 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.367205    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.367980 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.367961    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.425203 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.425176    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnw5p\" (UniqueName: \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\") pod \"payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.450965 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.450942    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:16.821955 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.821931    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 20:59:16.823910 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:16.823881    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e6f36f_7e24_4485_8236_3b351d9f75ed.slice/crio-c7671258d708b5765d2bced9b0d5050418e4a65209b2a627af6c1258d7642253 WatchSource:0}: Error finding container c7671258d708b5765d2bced9b0d5050418e4a65209b2a627af6c1258d7642253: Status 404 returned error can't find the container with id c7671258d708b5765d2bced9b0d5050418e4a65209b2a627af6c1258d7642253
Nov 03 20:59:16.825963 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:16.825942    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:59:17.552330 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:17.552290    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" event={"ID":"18e6f36f-7e24-4485-8236-3b351d9f75ed","Type":"ContainerStarted","Data":"c7671258d708b5765d2bced9b0d5050418e4a65209b2a627af6c1258d7642253"}
Nov 03 20:59:18.655434 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.655399    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 20:59:18.658986 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.658964    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.757475 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.757441    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-367d7739-dockercfg-49h9z\""
Nov 03 20:59:18.757475 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.757475    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-367d7739\""
Nov 03 20:59:18.760332 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.760299    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 20:59:18.793188 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.793151    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhfw\" (UniqueName: \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.793326 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.793266    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.805981 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.805958    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 20:59:18.809299 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.809283    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:18.848252 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.848225    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 20:59:18.894755 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.894733    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.894863 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.894766    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:18.894863 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.894804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhfw\" (UniqueName: \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.894863 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.894856    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfh4s\" (UniqueName: \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:18.895380 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.895362    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.952204 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.952175    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhfw\" (UniqueName: \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\") pod \"payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.971963 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.971938    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:18.997457 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.996184    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfh4s\" (UniqueName: \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:18.997457 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.996261    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:18.997457 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:18.997157    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:19.080524 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.080487    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfh4s\" (UniqueName: \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\") pod \"payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") " pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:19.123057 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.123028    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:19.267518 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:59:19.267437    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:59:19.371766 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.371740    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 20:59:19.374396 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:19.374366    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756e3313_1a04_4cc9_972e_293dd1972d3d.slice/crio-1abcd3ab716ab23901751577190703e5741cd2a86a227ff96b4ad8b34af58193 WatchSource:0}: Error finding container 1abcd3ab716ab23901751577190703e5741cd2a86a227ff96b4ad8b34af58193: Status 404 returned error can't find the container with id 1abcd3ab716ab23901751577190703e5741cd2a86a227ff96b4ad8b34af58193
Nov 03 20:59:19.493240 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.493216    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 20:59:19.495166 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:19.495137    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b9846b_14b0_4409_a31e_9737a3ccaaec.slice/crio-e44587e51d05ab1145e5687a989de176b2c9a42e7b36f992321c2f2a8a80a85f WatchSource:0}: Error finding container e44587e51d05ab1145e5687a989de176b2c9a42e7b36f992321c2f2a8a80a85f: Status 404 returned error can't find the container with id e44587e51d05ab1145e5687a989de176b2c9a42e7b36f992321c2f2a8a80a85f
Nov 03 20:59:19.562623 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.562550    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" event={"ID":"756e3313-1a04-4cc9-972e-293dd1972d3d","Type":"ContainerStarted","Data":"1abcd3ab716ab23901751577190703e5741cd2a86a227ff96b4ad8b34af58193"}
Nov 03 20:59:19.563858 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:19.563834    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" event={"ID":"c2b9846b-14b0-4409-a31e-9737a3ccaaec","Type":"ContainerStarted","Data":"e44587e51d05ab1145e5687a989de176b2c9a42e7b36f992321c2f2a8a80a85f"}
Nov 03 20:59:21.422301 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.422260    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 20:59:21.426351 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.426326    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.479245 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.479219    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 20:59:21.523508 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.523482    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b32d5a5\""
Nov 03 20:59:21.523627 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.523490    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b32d5a5-dockercfg-95zhg\""
Nov 03 20:59:21.528896 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.528873    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.528997 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.528935    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xkx\" (UniqueName: \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.576266 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.576235    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" event={"ID":"18e6f36f-7e24-4485-8236-3b351d9f75ed","Type":"ContainerStarted","Data":"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"}
Nov 03 20:59:21.576397 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.576339    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:21.629742 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.629712    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.629858 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.629771    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xkx\" (UniqueName: \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.630398 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.630375    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.688107 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.688028    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xkx\" (UniqueName: \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\") pod \"payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") " pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:21.743426 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:21.743400    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:22.106210 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.105854    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" podStartSLOduration=2.108329953 podStartE2EDuration="6.105835509s" totalImagesPullingTime="3.997505556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:21.692703056 +0000 UTC m=+663.324016534" watchObservedRunningTime="2026-11-03 20:59:22.105835509 +0000 UTC m=+663.737148987"
Nov 03 20:59:22.106781 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.106752    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 20:59:22.108768 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:22.108740    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode568fc2d_be98_44aa_8bca_b9e0d021668c.slice/crio-795e93a848d17479303bc6f5b48bfd83531835a04a4fa250cf85412a15822674 WatchSource:0}: Error finding container 795e93a848d17479303bc6f5b48bfd83531835a04a4fa250cf85412a15822674: Status 404 returned error can't find the container with id 795e93a848d17479303bc6f5b48bfd83531835a04a4fa250cf85412a15822674
Nov 03 20:59:22.584175 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.584135    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" event={"ID":"e568fc2d-be98-44aa-8bca-b9e0d021668c","Type":"ContainerStarted","Data":"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"}
Nov 03 20:59:22.584794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.584206    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:22.584794 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.584230    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" event={"ID":"e568fc2d-be98-44aa-8bca-b9e0d021668c","Type":"ContainerStarted","Data":"795e93a848d17479303bc6f5b48bfd83531835a04a4fa250cf85412a15822674"}
Nov 03 20:59:22.697171 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:22.697122    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" podStartSLOduration=1.697107513 podStartE2EDuration="1.697107513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:22.695757317 +0000 UTC m=+664.327070816" watchObservedRunningTime="2026-11-03 20:59:22.697107513 +0000 UTC m=+664.328420987"
Nov 03 20:59:23.593533 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.593496    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" event={"ID":"756e3313-1a04-4cc9-972e-293dd1972d3d","Type":"ContainerStarted","Data":"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"}
Nov 03 20:59:23.593964 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.593651    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:23.595183 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.595157    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" event={"ID":"c2b9846b-14b0-4409-a31e-9737a3ccaaec","Type":"ContainerStarted","Data":"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"}
Nov 03 20:59:23.595315 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.595203    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:23.706118 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.706063    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" podStartSLOduration=2.452147352 podStartE2EDuration="5.706048178s" totalImagesPullingTime="3.253900826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:23.704603062 +0000 UTC m=+665.335916540" watchObservedRunningTime="2026-11-03 20:59:23.706048178 +0000 UTC m=+665.337361654"
Nov 03 20:59:23.815579 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:23.815527    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" podStartSLOduration=2.173070008 podStartE2EDuration="5.815514381s" totalImagesPullingTime="3.642444373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:23.814306538 +0000 UTC m=+665.445620016" watchObservedRunningTime="2026-11-03 20:59:23.815514381 +0000 UTC m=+665.446827902"
Nov 03 20:59:26.809319 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:26.809281    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:59:28.585290 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.585247    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 20:59:28.601399 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.601366    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.637652 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.637611    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 20:59:28.682029 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.681972    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c9f7f443\""
Nov 03 20:59:28.682207 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.682061    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c9f7f443-dockercfg-pqdbh\""
Nov 03 20:59:28.703901 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.703860    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcchf\" (UniqueName: \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.704263 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.704240    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.730523 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.730494    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 20:59:28.734009 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.733991    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.786386 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.786350    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 20:59:28.804926 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.804891    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcchf\" (UniqueName: \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.805099 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.804953    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.805099 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.804995    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.805099 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.805016    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946w9\" (UniqueName: \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.805851 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.805831    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.882424 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.882353    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcchf\" (UniqueName: \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\") pod \"payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.906522 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.906494    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.906673 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.906539    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946w9\" (UniqueName: \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.907179 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.907159    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:28.914555 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.914535    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:28.982671 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:28.981348    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946w9\" (UniqueName: \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\") pod \"payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") " pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:29.046126 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.046094    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:29.312076 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.312050    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 20:59:29.313629 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:29.313596    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0749c837_2981_4dec_be3d_703be6cb0825.slice/crio-518c7848583a9753ed71ced2184e622fe30d512e80e52c523311b5759021c5c7 WatchSource:0}: Error finding container 518c7848583a9753ed71ced2184e622fe30d512e80e52c523311b5759021c5c7: Status 404 returned error can't find the container with id 518c7848583a9753ed71ced2184e622fe30d512e80e52c523311b5759021c5c7
Nov 03 20:59:29.438559 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.438533    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 20:59:29.440200 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:29.440177    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53787f3d_5d7c_429c_a104_a5c9e11e14fc.slice/crio-efbe71451fa328a9b11c94181e255d952a56050c9fb5201877a01c72bfb2efb5 WatchSource:0}: Error finding container efbe71451fa328a9b11c94181e255d952a56050c9fb5201877a01c72bfb2efb5: Status 404 returned error can't find the container with id efbe71451fa328a9b11c94181e255d952a56050c9fb5201877a01c72bfb2efb5
Nov 03 20:59:29.627018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.626933    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" event={"ID":"0749c837-2981-4dec-be3d-703be6cb0825","Type":"ContainerStarted","Data":"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"}
Nov 03 20:59:29.627018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.626994    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:29.627018 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.627012    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" event={"ID":"0749c837-2981-4dec-be3d-703be6cb0825","Type":"ContainerStarted","Data":"518c7848583a9753ed71ced2184e622fe30d512e80e52c523311b5759021c5c7"}
Nov 03 20:59:29.628565 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.628537    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" event={"ID":"53787f3d-5d7c-429c-a104-a5c9e11e14fc","Type":"ContainerStarted","Data":"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"}
Nov 03 20:59:29.628677 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.628588    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" event={"ID":"53787f3d-5d7c-429c-a104-a5c9e11e14fc","Type":"ContainerStarted","Data":"efbe71451fa328a9b11c94181e255d952a56050c9fb5201877a01c72bfb2efb5"}
Nov 03 20:59:29.628677 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.628626    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:29.736661 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.736600    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" podStartSLOduration=1.7365849880000002 podStartE2EDuration="1.736584988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:29.735826425 +0000 UTC m=+671.367139903" watchObservedRunningTime="2026-11-03 20:59:29.736584988 +0000 UTC m=+671.367898478"
Nov 03 20:59:29.845891 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:29.845833    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" podStartSLOduration=1.845818293 podStartE2EDuration="1.845818293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:29.844659484 +0000 UTC m=+671.475972964" watchObservedRunningTime="2026-11-03 20:59:29.845818293 +0000 UTC m=+671.477131774"
Nov 03 20:59:32.587970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:32.587940    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 20:59:33.600270 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:33.600174    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 20:59:33.698089 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:33.698062    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:59:34.603946 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:34.603915    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 20:59:34.604432 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:34.603998    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 20:59:36.697036 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.697001    5533 generic.go:356] "Generic (PLEG): container finished" podID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerID="4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772" exitCode=0
Nov 03 20:59:36.697614 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.697112    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerDied","Data":"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"}
Nov 03 20:59:36.697614 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.697471    5533 scope.go:122] "RemoveContainer" containerID="4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"
Nov 03 20:59:36.698813 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.698784    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:36.698897 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.698852    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:36.698897 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:36.698891    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:37.708759 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:37.708725    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerStarted","Data":"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"}
Nov 03 20:59:37.720481 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:37.720449    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:37.721515 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:37.721493    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:37.721613 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:37.721536    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:38.508970 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:38.508940    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:38.720418 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:38.720386    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:38.720852 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:38.720445    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:39.720165 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:39.720126    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:39.720366 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:39.720181    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:40.637560 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:40.637530    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 20:59:40.638037 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:40.637598    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 20:59:40.720472 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:40.720437    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:40.720702 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:40.720506    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:41.720524 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:41.720487    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:41.720945 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:41.720544    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:42.720094 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:42.720044    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:59:42.720274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:42.720133    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 20:59:43.724129 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:43.724105    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:43.726128 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:43.726107    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 20:59:44.750545 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.750512    5533 generic.go:356] "Generic (PLEG): container finished" podID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerID="3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d" exitCode=0
Nov 03 20:59:44.750954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.750582    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerDied","Data":"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"}
Nov 03 20:59:44.750954 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.750916    5533 scope.go:122] "RemoveContainer" containerID="3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"
Nov 03 20:59:44.752063 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.752032    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:44.752181 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.752117    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:44.752181 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:44.752164    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:59:45.783509 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:45.783470    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerStarted","Data":"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"}
Nov 03 20:59:46.291354 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:46.291322    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:46.291530 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:46.291509    5533 patch_prober.go:29] interesting pod/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:59:46.291583 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:46.291557    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 20:59:47.291976 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:47.291938    5533 patch_prober.go:29] interesting pod/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:59:47.292362 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:47.291994    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 20:59:48.291604 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:48.291567    5533 patch_prober.go:29] interesting pod/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:59:48.291791 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:48.291620    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 20:59:49.280708 ip-10-0-140-144 kubenswrapper[5533]: E1103 20:59:49.280672    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 20:59:49.292124 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.292094    5533 patch_prober.go:29] interesting pod/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:59:49.292239 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.292148    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 20:59:49.579363 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.579274    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"]
Nov 03 20:59:49.583340 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.583322    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.679229 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.679194    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:59:49.679229 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.679227    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 20:59:49.681289 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.681263    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"]
Nov 03 20:59:49.681673 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.681656    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.681760 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.681745    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7t8\" (UniqueName: \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.681809 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.681802    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.783150 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.783121    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7t8\" (UniqueName: \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.783285 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.783163    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.783285 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.783230    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.783912 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.783889    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.785545 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.785524    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.840710 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.840631    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7t8\" (UniqueName: \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\") pod \"maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:49.896092 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:49.896067    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:50.264615 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:50.261088    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"]
Nov 03 20:59:50.286535 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:50.286488    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7c0744_b0bf_4536_97cc_335a773355c8.slice/crio-f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2 WatchSource:0}: Error finding container f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2: Status 404 returned error can't find the container with id f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2
Nov 03 20:59:50.291750 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:50.291726    5533 patch_prober.go:29] interesting pod/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:59:50.291864 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:50.291770    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 20:59:50.813701 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:50.813655    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd" event={"ID":"2b7c0744-b0bf-4536-97cc-335a773355c8","Type":"ContainerStarted","Data":"f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2"}
Nov 03 20:59:51.261265 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:51.261232    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 20:59:51.295980 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:51.295955    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:51.296392 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:51.296226    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:51.296838 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:51.296818    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 20:59:55.242796 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.242768    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"]
Nov 03 20:59:55.246404 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.246384    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.302516 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.302475    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"]
Nov 03 20:59:55.334133 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.334054    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.334133 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.334095    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.334414 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.334287    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtcn\" (UniqueName: \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.435719 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.435675    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.435920 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.435730    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.435920 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.435807    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtcn\" (UniqueName: \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.436413 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.436381    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.438274 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.438239    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.493422 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.493386    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtcn\" (UniqueName: \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\") pod \"maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.560449 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.560418    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:55.841248 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.841217    5533 generic.go:356] "Generic (PLEG): container finished" podID="2b7c0744-b0bf-4536-97cc-335a773355c8" containerID="ac416411dae7a72f4c89b5c2d16025cdd6b88a50508ccd4b7c1bac208a11ead2" exitCode=0
Nov 03 20:59:55.841391 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.841266    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd" event={"ID":"2b7c0744-b0bf-4536-97cc-335a773355c8","Type":"ContainerDied","Data":"ac416411dae7a72f4c89b5c2d16025cdd6b88a50508ccd4b7c1bac208a11ead2"}
Nov 03 20:59:55.925107 ip-10-0-140-144 kubenswrapper[5533]: W1103 20:59:55.925062    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ace246_dfa5_4437_9223_0a1d1d8ddfe3.slice/crio-8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032 WatchSource:0}: Error finding container 8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032: Status 404 returned error can't find the container with id 8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032
Nov 03 20:59:55.939327 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:55.939303    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"]
Nov 03 20:59:56.848556 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:56.848519    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d" event={"ID":"55ace246-dfa5-4437-9223-0a1d1d8ddfe3","Type":"ContainerStarted","Data":"68b3827f3fbafb77a3aef70b8c0a00e52ca144ca17382c36512604d5e5478d61"}
Nov 03 20:59:56.848990 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:56.848578    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d" event={"ID":"55ace246-dfa5-4437-9223-0a1d1d8ddfe3","Type":"ContainerStarted","Data":"8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032"}
Nov 03 20:59:56.961313 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:56.961262    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d" podStartSLOduration=1.961247497 podStartE2EDuration="1.961247497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:56.960020677 +0000 UTC m=+698.591334155" watchObservedRunningTime="2026-11-03 20:59:56.961247497 +0000 UTC m=+698.592560974"
Nov 03 20:59:56.989812 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:56.989783    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:57.150844 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.150781    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\") pod \"2b7c0744-b0bf-4536-97cc-335a773355c8\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") "
Nov 03 20:59:57.150979 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.150867    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\") pod \"2b7c0744-b0bf-4536-97cc-335a773355c8\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") "
Nov 03 20:59:57.150979 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.150918    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\" (UniqueName: \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\") pod \"2b7c0744-b0bf-4536-97cc-335a773355c8\" (UID: \"2b7c0744-b0bf-4536-97cc-335a773355c8\") "
Nov 03 20:59:57.151227 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.151204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca" pod "2b7c0744-b0bf-4536-97cc-335a773355c8" (UID: "2b7c0744-b0bf-4536-97cc-335a773355c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:57.153013 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.152985    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca" pod "2b7c0744-b0bf-4536-97cc-335a773355c8" (UID: "2b7c0744-b0bf-4536-97cc-335a773355c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:57.153119 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.153086    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8" pod "2b7c0744-b0bf-4536-97cc-335a773355c8" (UID: "2b7c0744-b0bf-4536-97cc-335a773355c8"). InnerVolumeSpecName "kube-api-access-8p7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:57.251791 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.251760    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:57.251791 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.251793    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7c0744-b0bf-4536-97cc-335a773355c8-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:57.251962 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.251809    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7t8\" (UniqueName: \"kubernetes.io/projected/2b7c0744-b0bf-4536-97cc-335a773355c8-kube-api-access-8p7t8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:57.855154 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.855128    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"
Nov 03 20:59:57.855606 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.855163    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd" event={"ID":"2b7c0744-b0bf-4536-97cc-335a773355c8","Type":"ContainerDied","Data":"f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2"}
Nov 03 20:59:57.855606 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.855216    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f412bc53966e2d1b717fcf6c0e7e0ce9f8f8c075610dd9ce8b320b26f5880bc2"
Nov 03 20:59:57.856869 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.856842    5533 generic.go:356] "Generic (PLEG): container finished" podID="55ace246-dfa5-4437-9223-0a1d1d8ddfe3" containerID="68b3827f3fbafb77a3aef70b8c0a00e52ca144ca17382c36512604d5e5478d61" exitCode=0
Nov 03 20:59:57.856978 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:57.856913    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d" event={"ID":"55ace246-dfa5-4437-9223-0a1d1d8ddfe3","Type":"ContainerDied","Data":"68b3827f3fbafb77a3aef70b8c0a00e52ca144ca17382c36512604d5e5478d61"}
Nov 03 20:59:58.992788 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:58.992767    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 20:59:59.167961 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.167883    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\" (UniqueName: \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\") pod \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") "
Nov 03 20:59:59.167961 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.167939    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\") pod \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") "
Nov 03 20:59:59.168180 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.168033    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\") pod \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\" (UID: \"55ace246-dfa5-4437-9223-0a1d1d8ddfe3\") "
Nov 03 20:59:59.168384 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.168348    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca" pod "55ace246-dfa5-4437-9223-0a1d1d8ddfe3" (UID: "55ace246-dfa5-4437-9223-0a1d1d8ddfe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:59.169989 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.169961    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn" pod "55ace246-dfa5-4437-9223-0a1d1d8ddfe3" (UID: "55ace246-dfa5-4437-9223-0a1d1d8ddfe3"). InnerVolumeSpecName "kube-api-access-djtcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:59.170140 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.170051    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca" pod "55ace246-dfa5-4437-9223-0a1d1d8ddfe3" (UID: "55ace246-dfa5-4437-9223-0a1d1d8ddfe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:59.268823 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.268800    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.268823 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.268821    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtcn\" (UniqueName: \"kubernetes.io/projected/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-kube-api-access-djtcn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.269035 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.268832    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ace246-dfa5-4437-9223-0a1d1d8ddfe3-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.869043 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.869011    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d" event={"ID":"55ace246-dfa5-4437-9223-0a1d1d8ddfe3","Type":"ContainerDied","Data":"8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032"}
Nov 03 20:59:59.869214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.869077    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a55390992efedfd01b4876fe2b10683a0d4073be3c9b1e785675b8b96a7a032"
Nov 03 20:59:59.869214 ip-10-0-140-144 kubenswrapper[5533]: I1103 20:59:59.869027    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"
Nov 03 21:00:00.166519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166441    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:00.166978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166869    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2b7c0744-b0bf-4536-97cc-335a773355c8" containerName="revoke-keys"
Nov 03 21:00:00.166978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166883    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7c0744-b0bf-4536-97cc-335a773355c8" containerName="revoke-keys"
Nov 03 21:00:00.166978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166911    5533 cpu_manager.go:414] "containerMap: removing container" podUID="55ace246-dfa5-4437-9223-0a1d1d8ddfe3" containerName="revoke-keys"
Nov 03 21:00:00.166978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166917    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ace246-dfa5-4437-9223-0a1d1d8ddfe3" containerName="revoke-keys"
Nov 03 21:00:00.167180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.166992    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7c0744-b0bf-4536-97cc-335a773355c8" containerName="revoke-keys"
Nov 03 21:00:00.167180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.167004    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ace246-dfa5-4437-9223-0a1d1d8ddfe3" containerName="revoke-keys"
Nov 03 21:00:00.170232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.170210    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:00.170369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.170342    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.173559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.173542    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:00.173701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.173683    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.176738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.176718    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:00.176859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.176831    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.179897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.179878    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.220001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.219981    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:00:00.260715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.260682    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:00.263999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.263979    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:00.277477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277446    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.277590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6psg\" (UniqueName: \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.277590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277511    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.277732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277663    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxkkn\" (UniqueName: \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.277732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277691    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlb5\" (UniqueName: \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.277732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277712    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9xs\" (UniqueName: \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.277893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277802    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.277893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.277820    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.307467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.307445    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:00.322447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.322426    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:00.378509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378485    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6psg\" (UniqueName: \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.378607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378518    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.378607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378568    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxkkn\" (UniqueName: \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.378757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378603    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlb5\" (UniqueName: \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.378757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378739    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9xs\" (UniqueName: \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.378874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378847    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.378932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378880    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.378932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.378918    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.379308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.379274    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.379516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.379497    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.379669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.379630    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.379708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.379630    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.437221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.437195    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6psg\" (UniqueName: \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\") pod \"maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.437325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.437217    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlb5\" (UniqueName: \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\") pod \"maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.437792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.437771    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxkkn\" (UniqueName: \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\") pod \"maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.438211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.438188    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9xs\" (UniqueName: \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\") pod \"maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.485985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.485962    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:00.491713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.491625    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:00.501528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.501504    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:00.508572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.508398    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:00.876474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.876449    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:00.877933 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:00.877906    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c00d77_1af9_4270_afad_21ed3a427a0e.slice/crio-323c05a03024c07f836c0e24f0cfe6d370336868719145efe8b7f84825153be4 WatchSource:0}: Error finding container 323c05a03024c07f836c0e24f0cfe6d370336868719145efe8b7f84825153be4: Status 404 returned error can't find the container with id 323c05a03024c07f836c0e24f0cfe6d370336868719145efe8b7f84825153be4
Nov 03 21:00:00.921315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.921294    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:00.923150 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:00.923124    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode280cdba_d6ad_4631_b88e_24253626e62f.slice/crio-2d47ba1bafc0ee9f5b1cb4f7616a24e11b6a72f128d086ae75b50571776d9b87 WatchSource:0}: Error finding container 2d47ba1bafc0ee9f5b1cb4f7616a24e11b6a72f128d086ae75b50571776d9b87: Status 404 returned error can't find the container with id 2d47ba1bafc0ee9f5b1cb4f7616a24e11b6a72f128d086ae75b50571776d9b87
Nov 03 21:00:00.966207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.966185    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:00.967682 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:00.967660    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4a8326_163f_483e_bd79_027e1e7f543a.slice/crio-7e262340b872c31989ca3e92d7fe44ee28dc8559b85096ac21832460844d0706 WatchSource:0}: Error finding container 7e262340b872c31989ca3e92d7fe44ee28dc8559b85096ac21832460844d0706: Status 404 returned error can't find the container with id 7e262340b872c31989ca3e92d7fe44ee28dc8559b85096ac21832460844d0706
Nov 03 21:00:00.969882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:00.969857    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:01.053357 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:01.053330    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1981d45_30e7_4b06_8902_a8b45df83576.slice/crio-10840545a9fb0d65ca9e3f0f60d60c65d07598399bb6c773451765caae4c0f3d WatchSource:0}: Error finding container 10840545a9fb0d65ca9e3f0f60d60c65d07598399bb6c773451765caae4c0f3d: Status 404 returned error can't find the container with id 10840545a9fb0d65ca9e3f0f60d60c65d07598399bb6c773451765caae4c0f3d
Nov 03 21:00:01.884857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.884832    5533 generic.go:356] "Generic (PLEG): container finished" podID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerID="9b70ad90bb1e32e6d67316b3fc62ec362653d30e002d0a72a56b1211a37d3d46" exitCode=60
Nov 03 21:00:01.885183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.884912    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerDied","Data":"9b70ad90bb1e32e6d67316b3fc62ec362653d30e002d0a72a56b1211a37d3d46"}
Nov 03 21:00:01.885183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.884977    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerStarted","Data":"323c05a03024c07f836c0e24f0cfe6d370336868719145efe8b7f84825153be4"}
Nov 03 21:00:01.885183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.885095    5533 scope.go:122] "RemoveContainer" containerID="9b70ad90bb1e32e6d67316b3fc62ec362653d30e002d0a72a56b1211a37d3d46"
Nov 03 21:00:01.886497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.886474    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" event={"ID":"b1981d45-30e7-4b06-8902-a8b45df83576","Type":"ContainerStarted","Data":"4bd16cf35e0e09c66b02a678d351f0450ea0b8ff5e765f611dfd2672d25ce6ef"}
Nov 03 21:00:01.886586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.886519    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" event={"ID":"b1981d45-30e7-4b06-8902-a8b45df83576","Type":"ContainerStarted","Data":"10840545a9fb0d65ca9e3f0f60d60c65d07598399bb6c773451765caae4c0f3d"}
Nov 03 21:00:01.888214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.888194    5533 generic.go:356] "Generic (PLEG): container finished" podID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerID="eae636bc804ad6e92f3d975b7497181e2f32ee312519ce28fba0bf5c028da681" exitCode=60
Nov 03 21:00:01.888307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.888274    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerDied","Data":"eae636bc804ad6e92f3d975b7497181e2f32ee312519ce28fba0bf5c028da681"}
Nov 03 21:00:01.888371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.888321    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerStarted","Data":"7e262340b872c31989ca3e92d7fe44ee28dc8559b85096ac21832460844d0706"}
Nov 03 21:00:01.888431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.888407    5533 scope.go:122] "RemoveContainer" containerID="eae636bc804ad6e92f3d975b7497181e2f32ee312519ce28fba0bf5c028da681"
Nov 03 21:00:01.889840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.889817    5533 generic.go:356] "Generic (PLEG): container finished" podID="e280cdba-d6ad-4631-b88e-24253626e62f" containerID="91e5ee7daa3c2ce9fc857387b86c97e43afe64bc39c9509de14929eef6daf9cd" exitCode=60
Nov 03 21:00:01.889932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.889865    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" event={"ID":"e280cdba-d6ad-4631-b88e-24253626e62f","Type":"ContainerDied","Data":"91e5ee7daa3c2ce9fc857387b86c97e43afe64bc39c9509de14929eef6daf9cd"}
Nov 03 21:00:01.889932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.889907    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" event={"ID":"e280cdba-d6ad-4631-b88e-24253626e62f","Type":"ContainerStarted","Data":"2d47ba1bafc0ee9f5b1cb4f7616a24e11b6a72f128d086ae75b50571776d9b87"}
Nov 03 21:00:01.890083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:01.890068    5533 scope.go:122] "RemoveContainer" containerID="91e5ee7daa3c2ce9fc857387b86c97e43afe64bc39c9509de14929eef6daf9cd"
Nov 03 21:00:02.208401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.208343    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" podStartSLOduration=2.208329487 podStartE2EDuration="2.208329487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.207755211 +0000 UTC m=+703.839068688" watchObservedRunningTime="2026-11-03 21:00:02.208329487 +0000 UTC m=+703.839642963"
Nov 03 21:00:02.802137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.802112    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"]
Nov 03 21:00:02.805500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.805482    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:02.856669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.856619    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:00:02.859417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.859384    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"]
Nov 03 21:00:02.898438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.898413    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1981d45-30e7-4b06-8902-a8b45df83576" containerID="4bd16cf35e0e09c66b02a678d351f0450ea0b8ff5e765f611dfd2672d25ce6ef" exitCode=60
Nov 03 21:00:02.898793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.898494    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" event={"ID":"b1981d45-30e7-4b06-8902-a8b45df83576","Type":"ContainerDied","Data":"4bd16cf35e0e09c66b02a678d351f0450ea0b8ff5e765f611dfd2672d25ce6ef"}
Nov 03 21:00:02.898793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.898717    5533 scope.go:122] "RemoveContainer" containerID="4bd16cf35e0e09c66b02a678d351f0450ea0b8ff5e765f611dfd2672d25ce6ef"
Nov 03 21:00:02.900362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.900339    5533 generic.go:356] "Generic (PLEG): container finished" podID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerID="b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8" exitCode=60
Nov 03 21:00:02.900456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.900418    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerDied","Data":"b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8"}
Nov 03 21:00:02.900515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.900473    5533 scope.go:122] "RemoveContainer" containerID="eae636bc804ad6e92f3d975b7497181e2f32ee312519ce28fba0bf5c028da681"
Nov 03 21:00:02.900779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.900712    5533 scope.go:122] "RemoveContainer" containerID="b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8"
Nov 03 21:00:02.900947 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:02.900923    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7_odh-ai-gateway-infra(4c4a8326-163f-483e-bd79-027e1e7f543a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a"
Nov 03 21:00:02.902340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.902318    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:02.902442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.902351    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g5b\" (UniqueName: \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:02.902487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.902447    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:02.902868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.902786    5533 generic.go:356] "Generic (PLEG): container finished" podID="e280cdba-d6ad-4631-b88e-24253626e62f" containerID="cb0918eed2db387b7123d75116ba655f1adfa9c60b70c7f718425241869a49a2" exitCode=60
Nov 03 21:00:02.902984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.902875    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" event={"ID":"e280cdba-d6ad-4631-b88e-24253626e62f","Type":"ContainerDied","Data":"cb0918eed2db387b7123d75116ba655f1adfa9c60b70c7f718425241869a49a2"}
Nov 03 21:00:02.903091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.903075    5533 scope.go:122] "RemoveContainer" containerID="cb0918eed2db387b7123d75116ba655f1adfa9c60b70c7f718425241869a49a2"
Nov 03 21:00:02.903272 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:02.903251    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc_odh-ai-gateway-infra(e280cdba-d6ad-4631-b88e-24253626e62f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" podUID="e280cdba-d6ad-4631-b88e-24253626e62f"
Nov 03 21:00:02.907574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.905220    5533 generic.go:356] "Generic (PLEG): container finished" podID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerID="10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689" exitCode=60
Nov 03 21:00:02.907574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.905253    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerDied","Data":"10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689"}
Nov 03 21:00:02.907574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.905527    5533 scope.go:122] "RemoveContainer" containerID="10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689"
Nov 03 21:00:02.907574 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:02.905714    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm_odh-ai-gateway-infra(68c00d77-1af9-4270-afad-21ed3a427a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e"
Nov 03 21:00:02.937242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.937225    5533 scope.go:122] "RemoveContainer" containerID="91e5ee7daa3c2ce9fc857387b86c97e43afe64bc39c9509de14929eef6daf9cd"
Nov 03 21:00:02.951972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:02.951952    5533 scope.go:122] "RemoveContainer" containerID="9b70ad90bb1e32e6d67316b3fc62ec362653d30e002d0a72a56b1211a37d3d46"
Nov 03 21:00:03.002969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.002909    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.002969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.002963    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g5b\" (UniqueName: \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.003392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.003369    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.003574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.003557    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.005363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.005340    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.060877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.060859    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g5b\" (UniqueName: \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\") pod \"maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.258180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.258093    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:03.626712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.626689    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"]
Nov 03 21:00:03.628419 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:03.628398    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0fc9a4_d31d_4c39_9494_2149fae03886.slice/crio-6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f WatchSource:0}: Error finding container 6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f: Status 404 returned error can't find the container with id 6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f
Nov 03 21:00:03.912479 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.912452    5533 scope.go:122] "RemoveContainer" containerID="10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689"
Nov 03 21:00:03.912852 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:03.912696    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm_odh-ai-gateway-infra(68c00d77-1af9-4270-afad-21ed3a427a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e"
Nov 03 21:00:03.913407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.913386    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh" event={"ID":"ff0fc9a4-d31d-4c39-9494-2149fae03886","Type":"ContainerStarted","Data":"6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f"}
Nov 03 21:00:03.915111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.915088    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1981d45-30e7-4b06-8902-a8b45df83576" containerID="fd19a6ccb6454ce1b58084b11c24260a02b5d3068da989dfe26776d1172df7c3" exitCode=60
Nov 03 21:00:03.915213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.915164    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" event={"ID":"b1981d45-30e7-4b06-8902-a8b45df83576","Type":"ContainerDied","Data":"fd19a6ccb6454ce1b58084b11c24260a02b5d3068da989dfe26776d1172df7c3"}
Nov 03 21:00:03.915213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.915206    5533 scope.go:122] "RemoveContainer" containerID="4bd16cf35e0e09c66b02a678d351f0450ea0b8ff5e765f611dfd2672d25ce6ef"
Nov 03 21:00:03.915407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.915393    5533 scope.go:122] "RemoveContainer" containerID="fd19a6ccb6454ce1b58084b11c24260a02b5d3068da989dfe26776d1172df7c3"
Nov 03 21:00:03.915588 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:03.915565    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn_odh-ai-gateway-infra(b1981d45-30e7-4b06-8902-a8b45df83576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" podUID="b1981d45-30e7-4b06-8902-a8b45df83576"
Nov 03 21:00:03.917061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.917040    5533 scope.go:122] "RemoveContainer" containerID="b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8"
Nov 03 21:00:03.917261 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:03.917233    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7_odh-ai-gateway-infra(4c4a8326-163f-483e-bd79-027e1e7f543a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a"
Nov 03 21:00:03.918668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:03.918652    5533 scope.go:122] "RemoveContainer" containerID="cb0918eed2db387b7123d75116ba655f1adfa9c60b70c7f718425241869a49a2"
Nov 03 21:00:03.918823 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:03.918808    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc_odh-ai-gateway-infra(e280cdba-d6ad-4631-b88e-24253626e62f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" podUID="e280cdba-d6ad-4631-b88e-24253626e62f"
Nov 03 21:00:04.013669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:04.013075    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 21:00:04.077928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:04.077890    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 21:00:04.925766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:04.925735    5533 generic.go:356] "Generic (PLEG): container finished" podID="ff0fc9a4-d31d-4c39-9494-2149fae03886" containerID="27ae5f2a74889f35ba74e90001e1c6225dfa7ab117bd723a4af0d706cfbf2d35" exitCode=0
Nov 03 21:00:04.926173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:04.925821    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh" event={"ID":"ff0fc9a4-d31d-4c39-9494-2149fae03886","Type":"ContainerDied","Data":"27ae5f2a74889f35ba74e90001e1c6225dfa7ab117bd723a4af0d706cfbf2d35"}
Nov 03 21:00:04.927689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:04.927671    5533 scope.go:122] "RemoveContainer" containerID="fd19a6ccb6454ce1b58084b11c24260a02b5d3068da989dfe26776d1172df7c3"
Nov 03 21:00:04.927877 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:04.927858    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn_odh-ai-gateway-infra(b1981d45-30e7-4b06-8902-a8b45df83576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" podUID="b1981d45-30e7-4b06-8902-a8b45df83576"
Nov 03 21:00:06.065039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.065014    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:06.132114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.132086    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\") pod \"ff0fc9a4-d31d-4c39-9494-2149fae03886\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") "
Nov 03 21:00:06.132256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.132129    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\") pod \"ff0fc9a4-d31d-4c39-9494-2149fae03886\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") "
Nov 03 21:00:06.132321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.132252    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\" (UniqueName: \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\") pod \"ff0fc9a4-d31d-4c39-9494-2149fae03886\" (UID: \"ff0fc9a4-d31d-4c39-9494-2149fae03886\") "
Nov 03 21:00:06.132443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.132422    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca" pod "ff0fc9a4-d31d-4c39-9494-2149fae03886" (UID: "ff0fc9a4-d31d-4c39-9494-2149fae03886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:06.132523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.132510    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:06.134100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.134080    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b" pod "ff0fc9a4-d31d-4c39-9494-2149fae03886" (UID: "ff0fc9a4-d31d-4c39-9494-2149fae03886"). InnerVolumeSpecName "kube-api-access-74g5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:06.134211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.134119    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca" pod "ff0fc9a4-d31d-4c39-9494-2149fae03886" (UID: "ff0fc9a4-d31d-4c39-9494-2149fae03886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:00:06.233863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.233809    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g5b\" (UniqueName: \"kubernetes.io/projected/ff0fc9a4-d31d-4c39-9494-2149fae03886-kube-api-access-74g5b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:06.233863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.233830    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff0fc9a4-d31d-4c39-9494-2149fae03886-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:06.698613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.698585    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 21:00:06.940016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.939989    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"
Nov 03 21:00:06.940173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.939990    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh" event={"ID":"ff0fc9a4-d31d-4c39-9494-2149fae03886","Type":"ContainerDied","Data":"6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f"}
Nov 03 21:00:06.940173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:06.940112    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b332b7483ba77bde012570fb1d72f512b9a0c28b2b545fef95cede9abdcba3f"
Nov 03 21:00:09.014078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.014030    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" podUID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" containerName="payload-processing" containerID="cri-o://9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262" gracePeriod=25
Nov 03 21:00:09.078629 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.078596    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" podUID="756e3313-1a04-4cc9-972e-293dd1972d3d" containerName="payload-pre-processing" containerID="cri-o://66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390" gracePeriod=25
Nov 03 21:00:09.323245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.323216    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 21:00:09.359048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.359019    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 21:00:09.359163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.359145    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\") pod \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") "
Nov 03 21:00:09.359264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.359254    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\" (UniqueName: \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\") pod \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\" (UID: \"c2b9846b-14b0-4409-a31e-9737a3ccaaec\") "
Nov 03 21:00:09.359485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.359467    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume" pod "c2b9846b-14b0-4409-a31e-9737a3ccaaec" (UID: "c2b9846b-14b0-4409-a31e-9737a3ccaaec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:09.361188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.361169    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s" pod "c2b9846b-14b0-4409-a31e-9737a3ccaaec" (UID: "c2b9846b-14b0-4409-a31e-9737a3ccaaec"). InnerVolumeSpecName "kube-api-access-pfh4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:09.459928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.459899    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\") pod \"756e3313-1a04-4cc9-972e-293dd1972d3d\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") "
Nov 03 21:00:09.460050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.459991    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\" (UniqueName: \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\") pod \"756e3313-1a04-4cc9-972e-293dd1972d3d\" (UID: \"756e3313-1a04-4cc9-972e-293dd1972d3d\") "
Nov 03 21:00:09.460200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.460186    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfh4s\" (UniqueName: \"kubernetes.io/projected/c2b9846b-14b0-4409-a31e-9737a3ccaaec-kube-api-access-pfh4s\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:09.460278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.460201    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b9846b-14b0-4409-a31e-9737a3ccaaec-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:09.460356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.460330    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume" pod "756e3313-1a04-4cc9-972e-293dd1972d3d" (UID: "756e3313-1a04-4cc9-972e-293dd1972d3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:09.461902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.461880    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw" pod "756e3313-1a04-4cc9-972e-293dd1972d3d" (UID: "756e3313-1a04-4cc9-972e-293dd1972d3d"). InnerVolumeSpecName "kube-api-access-vnhfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:09.561723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.561652    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhfw\" (UniqueName: \"kubernetes.io/projected/756e3313-1a04-4cc9-972e-293dd1972d3d-kube-api-access-vnhfw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:09.561723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.561680    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756e3313-1a04-4cc9-972e-293dd1972d3d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:09.916282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.916203    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:09.957509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.957478    5533 generic.go:356] "Generic (PLEG): container finished" podID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" containerID="9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262" exitCode=0
Nov 03 21:00:09.957688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.957522    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"
Nov 03 21:00:09.957688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.957565    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" event={"ID":"c2b9846b-14b0-4409-a31e-9737a3ccaaec","Type":"ContainerDied","Data":"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"}
Nov 03 21:00:09.957688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.957626    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd" event={"ID":"c2b9846b-14b0-4409-a31e-9737a3ccaaec","Type":"ContainerDied","Data":"e44587e51d05ab1145e5687a989de176b2c9a42e7b36f992321c2f2a8a80a85f"}
Nov 03 21:00:09.957688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.957680    5533 scope.go:122] "RemoveContainer" containerID="9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"
Nov 03 21:00:09.961899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.961869    5533 generic.go:356] "Generic (PLEG): container finished" podID="756e3313-1a04-4cc9-972e-293dd1972d3d" containerID="66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390" exitCode=0
Nov 03 21:00:09.962051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.962028    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" event={"ID":"756e3313-1a04-4cc9-972e-293dd1972d3d","Type":"ContainerDied","Data":"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"}
Nov 03 21:00:09.962118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.962085    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h" event={"ID":"756e3313-1a04-4cc9-972e-293dd1972d3d","Type":"ContainerDied","Data":"1abcd3ab716ab23901751577190703e5741cd2a86a227ff96b4ad8b34af58193"}
Nov 03 21:00:09.962187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.962163    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"
Nov 03 21:00:09.979234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.978395    5533 scope.go:122] "RemoveContainer" containerID="9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"
Nov 03 21:00:09.979234 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:09.978843    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262\": container with ID starting with 9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262 not found: ID does not exist" containerID="9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"
Nov 03 21:00:09.979234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.978958    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262"} err="failed to get container status \"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262\": rpc error: code = NotFound desc = could not find container \"9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262\": container with ID starting with 9e419a7486ae382430bc6606a2680e0cce589d1997b0df9c857afe6bad5e5262 not found: ID does not exist"
Nov 03 21:00:09.979234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.978988    5533 scope.go:122] "RemoveContainer" containerID="66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"
Nov 03 21:00:09.995170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.995128    5533 scope.go:122] "RemoveContainer" containerID="66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"
Nov 03 21:00:09.995596 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:09.995568    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390\": container with ID starting with 66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390 not found: ID does not exist" containerID="66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"
Nov 03 21:00:09.995730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:09.995606    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390"} err="failed to get container status \"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390\": rpc error: code = NotFound desc = could not find container \"66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390\": container with ID starting with 66c6fbde7a5db1f99df11eb542662aa8d8095c51f002d4fada2b6cc5d5718390 not found: ID does not exist"
Nov 03 21:00:10.075884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.075841    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:10.132680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.132627    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 21:00:10.140899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.140873    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-367d7739-5775d66f4d-lhzmd"]
Nov 03 21:00:10.169954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.169891    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\") pod \"b1981d45-30e7-4b06-8902-a8b45df83576\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") "
Nov 03 21:00:10.169954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.169935    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\" (UniqueName: \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\") pod \"b1981d45-30e7-4b06-8902-a8b45df83576\" (UID: \"b1981d45-30e7-4b06-8902-a8b45df83576\") "
Nov 03 21:00:10.170302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.170278    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca" pod "b1981d45-30e7-4b06-8902-a8b45df83576" (UID: "b1981d45-30e7-4b06-8902-a8b45df83576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:10.171871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.171843    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg" pod "b1981d45-30e7-4b06-8902-a8b45df83576" (UID: "b1981d45-30e7-4b06-8902-a8b45df83576"). InnerVolumeSpecName "kube-api-access-x6psg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:10.244117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.244090    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 21:00:10.248514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.248492    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-367d7739-77d8b4bfc5-8rf5h"]
Nov 03 21:00:10.271240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.271216    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1981d45-30e7-4b06-8902-a8b45df83576-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:10.271240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.271237    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6psg\" (UniqueName: \"kubernetes.io/projected/b1981d45-30e7-4b06-8902-a8b45df83576-kube-api-access-x6psg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:10.491747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.491717    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756e3313-1a04-4cc9-972e-293dd1972d3d" path="/var/lib/kubelet/pods/756e3313-1a04-4cc9-972e-293dd1972d3d/volumes"
Nov 03 21:00:10.492074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.492061    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" path="/var/lib/kubelet/pods/c2b9846b-14b0-4409-a31e-9737a3ccaaec/volumes"
Nov 03 21:00:10.968697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.968666    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"
Nov 03 21:00:10.968697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.968690    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn" event={"ID":"b1981d45-30e7-4b06-8902-a8b45df83576","Type":"ContainerDied","Data":"10840545a9fb0d65ca9e3f0f60d60c65d07598399bb6c773451765caae4c0f3d"}
Nov 03 21:00:10.968928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:10.968749    5533 scope.go:122] "RemoveContainer" containerID="fd19a6ccb6454ce1b58084b11c24260a02b5d3068da989dfe26776d1172df7c3"
Nov 03 21:00:11.067286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.067250    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:11.131220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.131185    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:11.138301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.138277    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-367d7739-29895660-2t5qn"]
Nov 03 21:00:11.199514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.199490    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:11.279872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.279817    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\") pod \"e280cdba-d6ad-4631-b88e-24253626e62f\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") "
Nov 03 21:00:11.279987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.279889    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\" (UniqueName: \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\") pod \"e280cdba-d6ad-4631-b88e-24253626e62f\" (UID: \"e280cdba-d6ad-4631-b88e-24253626e62f\") "
Nov 03 21:00:11.280142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.280118    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca" pod "e280cdba-d6ad-4631-b88e-24253626e62f" (UID: "e280cdba-d6ad-4631-b88e-24253626e62f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:11.281859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.281834    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn" pod "e280cdba-d6ad-4631-b88e-24253626e62f" (UID: "e280cdba-d6ad-4631-b88e-24253626e62f"). InnerVolumeSpecName "kube-api-access-xxkkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:11.381379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.381352    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e280cdba-d6ad-4631-b88e-24253626e62f-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:11.381379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.381377    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxkkn\" (UniqueName: \"kubernetes.io/projected/e280cdba-d6ad-4631-b88e-24253626e62f-kube-api-access-xxkkn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:11.699112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.699082    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" podUID="18e6f36f-7e24-4485-8236-3b351d9f75ed" containerName="payload-pre-processing" containerID="cri-o://d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d" gracePeriod=25
Nov 03 21:00:11.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.939441    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 21:00:11.973596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.973536    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"]
Nov 03 21:00:11.979966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.979941    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc" event={"ID":"e280cdba-d6ad-4631-b88e-24253626e62f","Type":"ContainerDied","Data":"2d47ba1bafc0ee9f5b1cb4f7616a24e11b6a72f128d086ae75b50571776d9b87"}
Nov 03 21:00:11.980066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.979962    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"
Nov 03 21:00:11.980066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.980004    5533 scope.go:122] "RemoveContainer" containerID="cb0918eed2db387b7123d75116ba655f1adfa9c60b70c7f718425241869a49a2"
Nov 03 21:00:11.981801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.981777    5533 generic.go:356] "Generic (PLEG): container finished" podID="18e6f36f-7e24-4485-8236-3b351d9f75ed" containerID="d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d" exitCode=0
Nov 03 21:00:11.981894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.981836    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"
Nov 03 21:00:11.981894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.981858    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" event={"ID":"18e6f36f-7e24-4485-8236-3b351d9f75ed","Type":"ContainerDied","Data":"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"}
Nov 03 21:00:11.981966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.981903    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc" event={"ID":"18e6f36f-7e24-4485-8236-3b351d9f75ed","Type":"ContainerDied","Data":"c7671258d708b5765d2bced9b0d5050418e4a65209b2a627af6c1258d7642253"}
Nov 03 21:00:11.986919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.986895    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\") pod \"18e6f36f-7e24-4485-8236-3b351d9f75ed\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") "
Nov 03 21:00:11.987040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.986950    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\" (UniqueName: \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\") pod \"18e6f36f-7e24-4485-8236-3b351d9f75ed\" (UID: \"18e6f36f-7e24-4485-8236-3b351d9f75ed\") "
Nov 03 21:00:11.987678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.987614    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume" pod "18e6f36f-7e24-4485-8236-3b351d9f75ed" (UID: "18e6f36f-7e24-4485-8236-3b351d9f75ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:11.989439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.989424    5533 scope.go:122] "RemoveContainer" containerID="d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"
Nov 03 21:00:11.989618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.989600    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p" pod "18e6f36f-7e24-4485-8236-3b351d9f75ed" (UID: "18e6f36f-7e24-4485-8236-3b351d9f75ed"). InnerVolumeSpecName "kube-api-access-gnw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:11.997854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.997797    5533 scope.go:122] "RemoveContainer" containerID="d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"
Nov 03 21:00:11.998086 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:11.998060    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d\": container with ID starting with d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d not found: ID does not exist" containerID="d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"
Nov 03 21:00:11.998150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:11.998085    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d"} err="failed to get container status \"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d\": rpc error: code = NotFound desc = could not find container \"d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d\": container with ID starting with d1d77bd5a0699cbfe6b57e37ab2b22fee19756830042a2de13a8f91be38b775d not found: ID does not exist"
Nov 03 21:00:12.020330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.020308    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-367d7739-zt2hd"]
Nov 03 21:00:12.066975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.066946    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 21:00:12.067199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.067163    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" containerID="cri-o://78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe" gracePeriod=30
Nov 03 21:00:12.088305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.088276    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18e6f36f-7e24-4485-8236-3b351d9f75ed-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:12.088424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.088311    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnw5p\" (UniqueName: \"kubernetes.io/projected/18e6f36f-7e24-4485-8236-3b351d9f75ed-kube-api-access-gnw5p\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:12.298408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.298378    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:12.304183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.304156    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0a30f61a-29895660-jmqtc"]
Nov 03 21:00:12.456436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.456411    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 21:00:12.460436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.460413    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0a30f61a-8588d4f8ff-h5ntc"]
Nov 03 21:00:12.492557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.492531    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e6f36f-7e24-4485-8236-3b351d9f75ed" path="/var/lib/kubelet/pods/18e6f36f-7e24-4485-8236-3b351d9f75ed/volumes"
Nov 03 21:00:12.492932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.492918    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7c0744-b0bf-4536-97cc-335a773355c8" path="/var/lib/kubelet/pods/2b7c0744-b0bf-4536-97cc-335a773355c8/volumes"
Nov 03 21:00:12.493244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.493232    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" path="/var/lib/kubelet/pods/b1981d45-30e7-4b06-8902-a8b45df83576/volumes"
Nov 03 21:00:12.493722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.493704    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" path="/var/lib/kubelet/pods/e280cdba-d6ad-4631-b88e-24253626e62f/volumes"
Nov 03 21:00:12.998215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:12.998185    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"]
Nov 03 21:00:13.000920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.000895    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a30f61a-mh67d"]
Nov 03 21:00:13.487101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.487072    5533 scope.go:122] "RemoveContainer" containerID="10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689"
Nov 03 21:00:13.726223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.726201    5533 patch_prober.go:29] interesting pod/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:00:13.726323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.726247    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:00:13.989982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.989947    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 21:00:13.995903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:13.995876    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerStarted","Data":"07ccc8b6e4bde8ddf6b1e06c5786418a24a2be1d40e3ab8f13a10f457b3e1474"}
Nov 03 21:00:14.210803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.210753    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" podStartSLOduration=14.210738337 podStartE2EDuration="14.210738337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:14.20999368 +0000 UTC m=+715.841307158" watchObservedRunningTime="2026-11-03 21:00:14.210738337 +0000 UTC m=+715.842051813"
Nov 03 21:00:14.449184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449152    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 21:00:14.449537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449525    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449538    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449557    5533 cpu_manager.go:414] "containerMap: removing container" podUID="756e3313-1a04-4cc9-972e-293dd1972d3d" containerName="payload-pre-processing"
Nov 03 21:00:14.449589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449562    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756e3313-1a04-4cc9-972e-293dd1972d3d" containerName="payload-pre-processing"
Nov 03 21:00:14.449589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449572    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449577    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449592    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" containerName="payload-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449597    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" containerName="payload-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449603    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449608    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449615    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449619    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449629    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ff0fc9a4-d31d-4c39-9494-2149fae03886" containerName="revoke-keys"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449647    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0fc9a4-d31d-4c39-9494-2149fae03886" containerName="revoke-keys"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449657    5533 cpu_manager.go:414] "containerMap: removing container" podUID="18e6f36f-7e24-4485-8236-3b351d9f75ed" containerName="payload-pre-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449662    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e6f36f-7e24-4485-8236-3b351d9f75ed" containerName="payload-pre-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449734    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449745    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="756e3313-1a04-4cc9-972e-293dd1972d3d" containerName="payload-pre-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449752    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1981d45-30e7-4b06-8902-a8b45df83576" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449759    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b9846b-14b0-4409-a31e-9737a3ccaaec" containerName="payload-processing"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449765    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449770    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e280cdba-d6ad-4631-b88e-24253626e62f" containerName="cleanup"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449776    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0fc9a4-d31d-4c39-9494-2149fae03886" containerName="revoke-keys"
Nov 03 21:00:14.449786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.449782    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e6f36f-7e24-4485-8236-3b351d9f75ed" containerName="payload-pre-processing"
Nov 03 21:00:14.452875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.452854    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.494323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.494284    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ace246-dfa5-4437-9223-0a1d1d8ddfe3" path="/var/lib/kubelet/pods/55ace246-dfa5-4437-9223-0a1d1d8ddfe3/volumes"
Nov 03 21:00:14.502018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.501993    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-099a5e-openshift-default-dockercfg-2h6nd\""
Nov 03 21:00:14.510873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.510811    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 21:00:14.513444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513409    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513466    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513564    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9fp\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513593    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513657    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513689    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513716    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513768    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.513895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.513789    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614684    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614727    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614751    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9fp\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614773    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614827    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.614869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614860    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614885    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614938    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.614963    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.615127    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.615271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.615402    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.615464    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.615775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.615754    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.617141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.617117    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.617622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.617601    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.673923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.673892    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.674814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.674786    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9fp\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:14.771437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:14.771356    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:15.002751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.002716    5533 generic.go:356] "Generic (PLEG): container finished" podID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerID="07ccc8b6e4bde8ddf6b1e06c5786418a24a2be1d40e3ab8f13a10f457b3e1474" exitCode=60
Nov 03 21:00:15.002901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.002796    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerDied","Data":"07ccc8b6e4bde8ddf6b1e06c5786418a24a2be1d40e3ab8f13a10f457b3e1474"}
Nov 03 21:00:15.002901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.002872    5533 scope.go:122] "RemoveContainer" containerID="10e06ef4351348ad8a8e2c58ffa139c415fa573cacf8bd2a491b493775568689"
Nov 03 21:00:15.003219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.003195    5533 scope.go:122] "RemoveContainer" containerID="07ccc8b6e4bde8ddf6b1e06c5786418a24a2be1d40e3ab8f13a10f457b3e1474"
Nov 03 21:00:15.003465 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:15.003448    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm_odh-ai-gateway-infra(68c00d77-1af9-4270-afad-21ed3a427a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e"
Nov 03 21:00:15.219962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.219925    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:15.243218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.243193    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 21:00:15.245082 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:15.245024    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b4b7fb_21d4_48ee_8d02_aa888260eff0.slice/crio-39599aa8ca2e4d067e0ec8a7055589d6816fcdfe9d73e1644aa3c7933548a869 WatchSource:0}: Error finding container 39599aa8ca2e4d067e0ec8a7055589d6816fcdfe9d73e1644aa3c7933548a869: Status 404 returned error can't find the container with id 39599aa8ca2e4d067e0ec8a7055589d6816fcdfe9d73e1644aa3c7933548a869
Nov 03 21:00:15.248212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.248176    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:15.248283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.248260    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:15.248328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.248303    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:15.969507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.969471    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"]
Nov 03 21:00:15.973847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:15.973825    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.009786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.009755    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerStarted","Data":"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"}
Nov 03 21:00:16.009898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.009818    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerStarted","Data":"39599aa8ca2e4d067e0ec8a7055589d6816fcdfe9d73e1644aa3c7933548a869"}
Nov 03 21:00:16.023388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.023361    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:00:16.025615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.025591    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"]
Nov 03 21:00:16.031074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.031047    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.031187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.031092    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhd42\" (UniqueName: \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.031256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.031185    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.132350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.132317    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhd42\" (UniqueName: \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.132506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.132396    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.132593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.132568    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.133288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.133263    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.134927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.134896    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.159924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.159900    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:16.189882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.189855    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhd42\" (UniqueName: \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\") pod \"maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.233890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.233835    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\" (UniqueName: \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\") pod \"68c00d77-1af9-4270-afad-21ed3a427a0e\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") "
Nov 03 21:00:16.234024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.234007    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\") pod \"68c00d77-1af9-4270-afad-21ed3a427a0e\" (UID: \"68c00d77-1af9-4270-afad-21ed3a427a0e\") "
Nov 03 21:00:16.234314 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.234293    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca" pod "68c00d77-1af9-4270-afad-21ed3a427a0e" (UID: "68c00d77-1af9-4270-afad-21ed3a427a0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:16.235747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.235723    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5" pod "68c00d77-1af9-4270-afad-21ed3a427a0e" (UID: "68c00d77-1af9-4270-afad-21ed3a427a0e"). InnerVolumeSpecName "kube-api-access-bmlb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:16.289953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.289924    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:16.308463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.308409    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podStartSLOduration=2.308392106 podStartE2EDuration="2.308392106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:16.306602322 +0000 UTC m=+717.937915799" watchObservedRunningTime="2026-11-03 21:00:16.308392106 +0000 UTC m=+717.939705584"
Nov 03 21:00:16.335081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.335058    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmlb5\" (UniqueName: \"kubernetes.io/projected/68c00d77-1af9-4270-afad-21ed3a427a0e-kube-api-access-bmlb5\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:16.335081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.335083    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c00d77-1af9-4270-afad-21ed3a427a0e-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:16.657065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.657037    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"]
Nov 03 21:00:16.658911 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:16.658874    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14d46e3_a86d_4d8b_8be6_f5d4e4d484f0.slice/crio-13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e WatchSource:0}: Error finding container 13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e: Status 404 returned error can't find the container with id 13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e
Nov 03 21:00:16.772197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.772135    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:16.776717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.776695    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:16.884138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:16.884115    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:00:17.019605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.019579    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"
Nov 03 21:00:17.020019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.019586    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm" event={"ID":"68c00d77-1af9-4270-afad-21ed3a427a0e","Type":"ContainerDied","Data":"323c05a03024c07f836c0e24f0cfe6d370336868719145efe8b7f84825153be4"}
Nov 03 21:00:17.020019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.019699    5533 scope.go:122] "RemoveContainer" containerID="07ccc8b6e4bde8ddf6b1e06c5786418a24a2be1d40e3ab8f13a10f457b3e1474"
Nov 03 21:00:17.021399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.021369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x" event={"ID":"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0","Type":"ContainerStarted","Data":"aa743326ade3a72e6b3157964f2e3f510ee39402f77e6340e2a86442dfb4213d"}
Nov 03 21:00:17.021550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.021426    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x" event={"ID":"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0","Type":"ContainerStarted","Data":"13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e"}
Nov 03 21:00:17.021723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.021703    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:17.022752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.022700    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:17.196940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.196899    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:17.199240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.199210    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-10bfccd7-29895660-pgbmm"]
Nov 03 21:00:17.293270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.293187    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x" podStartSLOduration=2.293167151 podStartE2EDuration="2.293167151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:17.290987061 +0000 UTC m=+718.922300538" watchObservedRunningTime="2026-11-03 21:00:17.293167151 +0000 UTC m=+718.924480629"
Nov 03 21:00:17.354575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.354552    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 21:00:17.447564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447528    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447595    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447654    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447752    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447812    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:17.447948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447851    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447900    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.447948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447939    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.448099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.447983    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.448099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\") pod \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\" (UID: \"d89c6e7a-3aec-4b48-8467-bb27be1240c9\") "
Nov 03 21:00:17.448099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448043    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:17.448099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448062    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:17.448295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448229    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:17.448345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448331    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:17.448449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448430    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.448536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448453    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.448536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448466    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.448536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448478    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.448536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.448494    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.450284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.450254    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:17.450418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.450400    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:00:17.450606 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.450592    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "kube-api-access-qxksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:17.450696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.450620    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy" pod "d89c6e7a-3aec-4b48-8467-bb27be1240c9" (UID: "d89c6e7a-3aec-4b48-8467-bb27be1240c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:17.486807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.486785    5533 scope.go:122] "RemoveContainer" containerID="b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8"
Nov 03 21:00:17.549435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.549370    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.549435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.549395    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.549435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.549411    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxksj\" (UniqueName: \"kubernetes.io/projected/d89c6e7a-3aec-4b48-8467-bb27be1240c9-kube-api-access-qxksj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:17.549435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:17.549425    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d89c6e7a-3aec-4b48-8467-bb27be1240c9-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.029206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.029175    5533 generic.go:356] "Generic (PLEG): container finished" podID="a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" containerID="aa743326ade3a72e6b3157964f2e3f510ee39402f77e6340e2a86442dfb4213d" exitCode=0
Nov 03 21:00:18.029594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.029255    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x" event={"ID":"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0","Type":"ContainerDied","Data":"aa743326ade3a72e6b3157964f2e3f510ee39402f77e6340e2a86442dfb4213d"}
Nov 03 21:00:18.031138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.031115    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerStarted","Data":"4267aa55470a082fa91282d9a23044a6801decb53ce37e1bf360a008b00ffde0"}
Nov 03 21:00:18.032664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.032624    5533 generic.go:356] "Generic (PLEG): container finished" podID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerID="78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe" exitCode=0
Nov 03 21:00:18.032754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.032682    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"
Nov 03 21:00:18.032754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.032721    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerDied","Data":"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"}
Nov 03 21:00:18.032860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.032769    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7" event={"ID":"d89c6e7a-3aec-4b48-8467-bb27be1240c9","Type":"ContainerDied","Data":"f7afbd6337cd660653a6a6b0a539fc92ce96176e868a4c6ea72af8299166dc1d"}
Nov 03 21:00:18.032860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.032791    5533 scope.go:122] "RemoveContainer" containerID="78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"
Nov 03 21:00:18.042851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.042834    5533 scope.go:122] "RemoveContainer" containerID="4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"
Nov 03 21:00:18.050761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.050738    5533 scope.go:122] "RemoveContainer" containerID="78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"
Nov 03 21:00:18.051037 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:18.051017    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe\": container with ID starting with 78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe not found: ID does not exist" containerID="78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"
Nov 03 21:00:18.051128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.051043    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe"} err="failed to get container status \"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe\": rpc error: code = NotFound desc = could not find container \"78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe\": container with ID starting with 78ded48acb2ed9f0518663f25fd4c922f1a9c913816976b24958ff06747128fe not found: ID does not exist"
Nov 03 21:00:18.051128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.051060    5533 scope.go:122] "RemoveContainer" containerID="4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"
Nov 03 21:00:18.051343 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:18.051289    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772\": container with ID starting with 4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772 not found: ID does not exist" containerID="4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"
Nov 03 21:00:18.051402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.051352    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772"} err="failed to get container status \"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772\": rpc error: code = NotFound desc = could not find container \"4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772\": container with ID starting with 4939a1a86f2b62698df26cc5cb17f909786b80f80262b2a0b277aee6d108b772 not found: ID does not exist"
Nov 03 21:00:18.258500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.258441    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" podStartSLOduration=18.258424621 podStartE2EDuration="18.258424621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:18.257312907 +0000 UTC m=+719.888626385" watchObservedRunningTime="2026-11-03 21:00:18.258424621 +0000 UTC m=+719.889738100"
Nov 03 21:00:18.363619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.363554    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 21:00:18.368006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.367981    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-367d7739-gw-openshift-default-78f548cb49-fgsq7"]
Nov 03 21:00:18.494327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.494297    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" path="/var/lib/kubelet/pods/68c00d77-1af9-4270-afad-21ed3a427a0e/volumes"
Nov 03 21:00:18.494738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.494722    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" path="/var/lib/kubelet/pods/d89c6e7a-3aec-4b48-8467-bb27be1240c9/volumes"
Nov 03 21:00:18.990569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:18.990526    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" podUID="e568fc2d-be98-44aa-8bca-b9e0d021668c" containerName="payload-processing" containerID="cri-o://2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0" gracePeriod=25
Nov 03 21:00:19.039751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.039720    5533 generic.go:356] "Generic (PLEG): container finished" podID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerID="4267aa55470a082fa91282d9a23044a6801decb53ce37e1bf360a008b00ffde0" exitCode=60
Nov 03 21:00:19.040090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.039786    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerDied","Data":"4267aa55470a082fa91282d9a23044a6801decb53ce37e1bf360a008b00ffde0"}
Nov 03 21:00:19.040090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.039842    5533 scope.go:122] "RemoveContainer" containerID="b87156d36eac808f7a4a0c780721dee7a030315f27852de0cee1ea5efe1879b8"
Nov 03 21:00:19.040557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.040376    5533 scope.go:122] "RemoveContainer" containerID="4267aa55470a082fa91282d9a23044a6801decb53ce37e1bf360a008b00ffde0"
Nov 03 21:00:19.040688 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:19.040597    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7_odh-ai-gateway-infra(4c4a8326-163f-483e-bd79-027e1e7f543a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a"
Nov 03 21:00:19.175313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.175281    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:19.267304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.264959    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\") pod \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") "
Nov 03 21:00:19.267304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.265125    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\") pod \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") "
Nov 03 21:00:19.267304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.265166    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\" (UniqueName: \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\") pod \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\" (UID: \"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0\") "
Nov 03 21:00:19.267304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.265787    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca" pod "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" (UID: "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.267304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.266906    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.271025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.270995    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42" pod "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" (UID: "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0"). InnerVolumeSpecName "kube-api-access-vhd42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:19.271584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.271556    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:19.277742 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:19.277533    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:00:19.280170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.280116    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca" pod "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" (UID: "a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:00:19.300736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.300705    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 21:00:19.368074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.368050    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\") pod \"e568fc2d-be98-44aa-8bca-b9e0d021668c\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") "
Nov 03 21:00:19.368165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.368092    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\" (UniqueName: \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\") pod \"e568fc2d-be98-44aa-8bca-b9e0d021668c\" (UID: \"e568fc2d-be98-44aa-8bca-b9e0d021668c\") "
Nov 03 21:00:19.368278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.368268    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhd42\" (UniqueName: \"kubernetes.io/projected/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-kube-api-access-vhd42\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.368318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.368281    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.368391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.368371    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume" pod "e568fc2d-be98-44aa-8bca-b9e0d021668c" (UID: "e568fc2d-be98-44aa-8bca-b9e0d021668c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.370017 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.369999    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx" pod "e568fc2d-be98-44aa-8bca-b9e0d021668c" (UID: "e568fc2d-be98-44aa-8bca-b9e0d021668c"). InnerVolumeSpecName "kube-api-access-j5xkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:19.468988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.468960    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e568fc2d-be98-44aa-8bca-b9e0d021668c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.468988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:19.468990    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5xkx\" (UniqueName: \"kubernetes.io/projected/e568fc2d-be98-44aa-8bca-b9e0d021668c-kube-api-access-j5xkx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.046973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.046945    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"
Nov 03 21:00:20.047343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.046964    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x" event={"ID":"a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0","Type":"ContainerDied","Data":"13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e"}
Nov 03 21:00:20.047343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.047046    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c89ad8099bb480794e1c73d16507b2e2b4ba407d4ab92c64c2d92e8742292e"
Nov 03 21:00:20.048446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.048420    5533 generic.go:356] "Generic (PLEG): container finished" podID="e568fc2d-be98-44aa-8bca-b9e0d021668c" containerID="2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0" exitCode=0
Nov 03 21:00:20.048565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.048465    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"
Nov 03 21:00:20.048565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.048505    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" event={"ID":"e568fc2d-be98-44aa-8bca-b9e0d021668c","Type":"ContainerDied","Data":"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"}
Nov 03 21:00:20.048565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.048547    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh" event={"ID":"e568fc2d-be98-44aa-8bca-b9e0d021668c","Type":"ContainerDied","Data":"795e93a848d17479303bc6f5b48bfd83531835a04a4fa250cf85412a15822674"}
Nov 03 21:00:20.048741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.048578    5533 scope.go:122] "RemoveContainer" containerID="2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"
Nov 03 21:00:20.060084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.060072    5533 scope.go:122] "RemoveContainer" containerID="2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"
Nov 03 21:00:20.060381 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:20.060353    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0\": container with ID starting with 2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0 not found: ID does not exist" containerID="2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"
Nov 03 21:00:20.060466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.060390    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0"} err="failed to get container status \"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0\": rpc error: code = NotFound desc = could not find container \"2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0\": container with ID starting with 2b5634b0262b825e59c77daf48be21be144448611df3f416af088c9730b0e7f0 not found: ID does not exist"
Nov 03 21:00:20.184610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.184588    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:20.203494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.203472    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 21:00:20.217538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.217518    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b32d5a5-7f77475896-gv4wh"]
Nov 03 21:00:20.275081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.275030    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\") pod \"4c4a8326-163f-483e-bd79-027e1e7f543a\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") "
Nov 03 21:00:20.275175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.275089    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\" (UniqueName: \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\") pod \"4c4a8326-163f-483e-bd79-027e1e7f543a\" (UID: \"4c4a8326-163f-483e-bd79-027e1e7f543a\") "
Nov 03 21:00:20.275453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.275429    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca" pod "4c4a8326-163f-483e-bd79-027e1e7f543a" (UID: "4c4a8326-163f-483e-bd79-027e1e7f543a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:20.277682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.277657    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs" pod "4c4a8326-163f-483e-bd79-027e1e7f543a" (UID: "4c4a8326-163f-483e-bd79-027e1e7f543a"). InnerVolumeSpecName "kube-api-access-gn9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:20.375829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.375804    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a8326-163f-483e-bd79-027e1e7f543a-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.375829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.375827    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9xs\" (UniqueName: \"kubernetes.io/projected/4c4a8326-163f-483e-bd79-027e1e7f543a-kube-api-access-gn9xs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.494224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:20.494201    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e568fc2d-be98-44aa-8bca-b9e0d021668c" path="/var/lib/kubelet/pods/e568fc2d-be98-44aa-8bca-b9e0d021668c/volumes"
Nov 03 21:00:21.060096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:21.060064    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"
Nov 03 21:00:21.060509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:21.060070    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7" event={"ID":"4c4a8326-163f-483e-bd79-027e1e7f543a","Type":"ContainerDied","Data":"7e262340b872c31989ca3e92d7fe44ee28dc8559b85096ac21832460844d0706"}
Nov 03 21:00:21.060509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:21.060208    5533 scope.go:122] "RemoveContainer" containerID="4267aa55470a082fa91282d9a23044a6801decb53ce37e1bf360a008b00ffde0"
Nov 03 21:00:21.214033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:21.214006    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:21.260558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:21.260535    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-5b32d5a5-29895660-84jc7"]
Nov 03 21:00:22.492209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:22.492177    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" path="/var/lib/kubelet/pods/4c4a8326-163f-483e-bd79-027e1e7f543a/volumes"
Nov 03 21:00:23.501992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.501965    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502313    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e568fc2d-be98-44aa-8bca-b9e0d021668c" containerName="payload-processing"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502324    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e568fc2d-be98-44aa-8bca-b9e0d021668c" containerName="payload-processing"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502335    5533 cpu_manager.go:414] "containerMap: removing container" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502340    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502347    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502352    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502359    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" containerName="revoke-keys"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502364    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" containerName="revoke-keys"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502385    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502390    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502397    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502402    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502409    5533 cpu_manager.go:414] "containerMap: removing container" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502423    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502429    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502433    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502488    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e568fc2d-be98-44aa-8bca-b9e0d021668c" containerName="payload-processing"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502494    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502500    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502505    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" containerName="revoke-keys"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502512    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502519    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502524    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89c6e7a-3aec-4b48-8467-bb27be1240c9" containerName="istio-proxy"
Nov 03 21:00:23.502970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.502529    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:23.510293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.510272    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.560786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.560760    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 21:00:23.594675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.594651    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-320f60\""
Nov 03 21:00:23.603489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.603466    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.603685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.603659    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxddh\" (UniqueName: \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.605117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.605098    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-320f60-dockercfg-kbvnx\""
Nov 03 21:00:23.704588 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.704559    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxddh\" (UniqueName: \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.704760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.704611    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.705306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.705286    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.762357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.762297    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxddh\" (UniqueName: \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\") pod \"payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:23.823369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:23.823337    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:24.188705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:24.188623    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 21:00:24.191061 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:24.191032    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185d0df5_0fb4_4686_bf57_1a792d10d203.slice/crio-473486ff6c09ec29d65437b4dc340a3ed55dc2326c10674a80e1e3a2f565ab43 WatchSource:0}: Error finding container 473486ff6c09ec29d65437b4dc340a3ed55dc2326c10674a80e1e3a2f565ab43: Status 404 returned error can't find the container with id 473486ff6c09ec29d65437b4dc340a3ed55dc2326c10674a80e1e3a2f565ab43
Nov 03 21:00:25.084700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.084662    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" event={"ID":"185d0df5-0fb4-4686-bf57-1a792d10d203","Type":"ContainerStarted","Data":"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"}
Nov 03 21:00:25.085083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.084720    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" event={"ID":"185d0df5-0fb4-4686-bf57-1a792d10d203","Type":"ContainerStarted","Data":"473486ff6c09ec29d65437b4dc340a3ed55dc2326c10674a80e1e3a2f565ab43"}
Nov 03 21:00:25.085083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.084827    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:25.196300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.196239    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" podStartSLOduration=2.196220117 podStartE2EDuration="2.196220117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:25.193181464 +0000 UTC m=+726.824494945" watchObservedRunningTime="2026-11-03 21:00:25.196220117 +0000 UTC m=+726.827533598"
Nov 03 21:00:25.910309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910277    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 21:00:25.910756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910736    5533 cpu_manager.go:414] "containerMap: removing container" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:25.910872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910757    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:25.910872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910788    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:25.910872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910796    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:25.911022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910919    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c00d77-1af9-4270-afad-21ed3a427a0e" containerName="cleanup"
Nov 03 21:00:25.911022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.910934    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4a8326-163f-483e-bd79-027e1e7f543a" containerName="cleanup"
Nov 03 21:00:25.914373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.914354    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:25.965562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.965538    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-099a5e\""
Nov 03 21:00:25.966087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.966062    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-099a5e-dockercfg-ghmcn\""
Nov 03 21:00:25.968366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:25.968344    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 21:00:26.024974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.024941    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.025142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.025049    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhb4b\" (UniqueName: \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.058749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.056184    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 21:00:26.063517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.063474    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.117330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.117235    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 21:00:26.126211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.126180    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.126211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.126221    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhb4b\" (UniqueName: \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.126440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.126293    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zz6\" (UniqueName: \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.126440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.126324    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.126930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.126906    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.184118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.184089    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhb4b\" (UniqueName: \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\") pod \"payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.226834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.226797    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zz6\" (UniqueName: \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.226945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.226847    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.227153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.227134    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:26.227582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.227564    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.287408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.287374    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zz6\" (UniqueName: \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\") pod \"payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.385825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.385789    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:26.597940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.597908    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 21:00:26.600273 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:26.600243    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd5d6ba_284b_4d43_b965_d55c418c4e13.slice/crio-ffd1ce9dfcae2ee686bb68494feaf2a9c4c4e4f8252e07f61497b214fd13c10c WatchSource:0}: Error finding container ffd1ce9dfcae2ee686bb68494feaf2a9c4c4e4f8252e07f61497b214fd13c10c: Status 404 returned error can't find the container with id ffd1ce9dfcae2ee686bb68494feaf2a9c4c4e4f8252e07f61497b214fd13c10c
Nov 03 21:00:26.600782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.600762    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 21:00:26.706997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.706946    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 21:00:26.749683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:26.749662    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 21:00:26.752367 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:26.752339    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c8fb54_cdba_4d31_872b_473b25eab3d9.slice/crio-44ea7b6b3e8b868e619c4fa76596430dbeae473725f55e875b6d543cad019134 WatchSource:0}: Error finding container 44ea7b6b3e8b868e619c4fa76596430dbeae473725f55e875b6d543cad019134: Status 404 returned error can't find the container with id 44ea7b6b3e8b868e619c4fa76596430dbeae473725f55e875b6d543cad019134
Nov 03 21:00:27.099018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.098928    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" event={"ID":"5dd5d6ba-284b-4d43-b965-d55c418c4e13","Type":"ContainerStarted","Data":"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"}
Nov 03 21:00:27.099166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.099021    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" event={"ID":"5dd5d6ba-284b-4d43-b965-d55c418c4e13","Type":"ContainerStarted","Data":"ffd1ce9dfcae2ee686bb68494feaf2a9c4c4e4f8252e07f61497b214fd13c10c"}
Nov 03 21:00:27.099166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.099053    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:27.100451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.100423    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" event={"ID":"22c8fb54-cdba-4d31-872b-473b25eab3d9","Type":"ContainerStarted","Data":"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"}
Nov 03 21:00:27.100554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.100463    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" event={"ID":"22c8fb54-cdba-4d31-872b-473b25eab3d9","Type":"ContainerStarted","Data":"44ea7b6b3e8b868e619c4fa76596430dbeae473725f55e875b6d543cad019134"}
Nov 03 21:00:27.100606 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.100549    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:27.209552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.209498    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" podStartSLOduration=2.209486023 podStartE2EDuration="2.209486023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:27.207843425 +0000 UTC m=+728.839156902" watchObservedRunningTime="2026-11-03 21:00:27.209486023 +0000 UTC m=+728.840799500"
Nov 03 21:00:27.318328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:27.318275    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" podStartSLOduration=1.318263 podStartE2EDuration="1.318263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:27.316030254 +0000 UTC m=+728.947343731" watchObservedRunningTime="2026-11-03 21:00:27.318263 +0000 UTC m=+728.949576488"
Nov 03 21:00:29.901696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:29.901598    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"]
Nov 03 21:00:29.948415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:29.948384    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b32d5a5-s57vh"]
Nov 03 21:00:30.492582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:30.492552    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0fc9a4-d31d-4c39-9494-2149fae03886" path="/var/lib/kubelet/pods/ff0fc9a4-d31d-4c39-9494-2149fae03886/volumes"
Nov 03 21:00:30.509869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:30.509847    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 21:00:30.510042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:30.510017    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy" containerID="cri-o://9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0" gracePeriod=30
Nov 03 21:00:31.601822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.601768    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" podUID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" containerName="payload-processing" containerID="cri-o://b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d" gracePeriod=25
Nov 03 21:00:31.708152 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.708079    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" podUID="0749c837-2981-4dec-be3d-703be6cb0825" containerName="payload-pre-processing" containerID="cri-o://0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463" gracePeriod=25
Nov 03 21:00:31.854836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.854768    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 21:00:31.980768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.980740    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 21:00:31.984924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.984899    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\") pod \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") "
Nov 03 21:00:31.985075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.984966    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\" (UniqueName: \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\") pod \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\" (UID: \"53787f3d-5d7c-429c-a104-a5c9e11e14fc\") "
Nov 03 21:00:31.985548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.985513    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume" pod "53787f3d-5d7c-429c-a104-a5c9e11e14fc" (UID: "53787f3d-5d7c-429c-a104-a5c9e11e14fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:31.987324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:31.987298    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9" pod "53787f3d-5d7c-429c-a104-a5c9e11e14fc" (UID: "53787f3d-5d7c-429c-a104-a5c9e11e14fc"). InnerVolumeSpecName "kube-api-access-946w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:32.085780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.085751    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\" (UniqueName: \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\") pod \"0749c837-2981-4dec-be3d-703be6cb0825\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") "
Nov 03 21:00:32.085961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.085828    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\") pod \"0749c837-2981-4dec-be3d-703be6cb0825\" (UID: \"0749c837-2981-4dec-be3d-703be6cb0825\") "
Nov 03 21:00:32.086191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.086171    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946w9\" (UniqueName: \"kubernetes.io/projected/53787f3d-5d7c-429c-a104-a5c9e11e14fc-kube-api-access-946w9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:32.086284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.086201    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53787f3d-5d7c-429c-a104-a5c9e11e14fc-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:32.086284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.086173    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume" pod "0749c837-2981-4dec-be3d-703be6cb0825" (UID: "0749c837-2981-4dec-be3d-703be6cb0825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:32.087826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.087804    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf" pod "0749c837-2981-4dec-be3d-703be6cb0825" (UID: "0749c837-2981-4dec-be3d-703be6cb0825"). InnerVolumeSpecName "kube-api-access-jcchf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:32.131911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.131833    5533 generic.go:356] "Generic (PLEG): container finished" podID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" containerID="b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d" exitCode=0
Nov 03 21:00:32.131911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.131869    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"
Nov 03 21:00:32.132083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.131911    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" event={"ID":"53787f3d-5d7c-429c-a104-a5c9e11e14fc","Type":"ContainerDied","Data":"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"}
Nov 03 21:00:32.132083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.131961    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5" event={"ID":"53787f3d-5d7c-429c-a104-a5c9e11e14fc","Type":"ContainerDied","Data":"efbe71451fa328a9b11c94181e255d952a56050c9fb5201877a01c72bfb2efb5"}
Nov 03 21:00:32.132083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.131985    5533 scope.go:122] "RemoveContainer" containerID="b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"
Nov 03 21:00:32.133382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.133354    5533 generic.go:356] "Generic (PLEG): container finished" podID="0749c837-2981-4dec-be3d-703be6cb0825" containerID="0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463" exitCode=0
Nov 03 21:00:32.133516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.133391    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"
Nov 03 21:00:32.133516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.133398    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" event={"ID":"0749c837-2981-4dec-be3d-703be6cb0825","Type":"ContainerDied","Data":"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"}
Nov 03 21:00:32.133516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.133460    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd" event={"ID":"0749c837-2981-4dec-be3d-703be6cb0825","Type":"ContainerDied","Data":"518c7848583a9753ed71ced2184e622fe30d512e80e52c523311b5759021c5c7"}
Nov 03 21:00:32.142412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.142394    5533 scope.go:122] "RemoveContainer" containerID="b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"
Nov 03 21:00:32.142699 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:32.142678    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d\": container with ID starting with b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d not found: ID does not exist" containerID="b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"
Nov 03 21:00:32.142776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.142705    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d"} err="failed to get container status \"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d\": rpc error: code = NotFound desc = could not find container \"b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d\": container with ID starting with b2d0e55571e87b9e49d59c5b37737333bbf25bb7ce5561713a7e2498e7fa213d not found: ID does not exist"
Nov 03 21:00:32.142776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.142723    5533 scope.go:122] "RemoveContainer" containerID="0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"
Nov 03 21:00:32.150722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.150697    5533 scope.go:122] "RemoveContainer" containerID="0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"
Nov 03 21:00:32.150960 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:32.150944    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463\": container with ID starting with 0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463 not found: ID does not exist" containerID="0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"
Nov 03 21:00:32.151008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.150967    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463"} err="failed to get container status \"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463\": rpc error: code = NotFound desc = could not find container \"0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463\": container with ID starting with 0cdefa7d052138d83d472ef9dc92316be71de3e015933f5852cbb76c053b8463 not found: ID does not exist"
Nov 03 21:00:32.187362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.187336    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcchf\" (UniqueName: \"kubernetes.io/projected/0749c837-2981-4dec-be3d-703be6cb0825-kube-api-access-jcchf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:32.187362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.187362    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0749c837-2981-4dec-be3d-703be6cb0825-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:32.289788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.289757    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 21:00:32.292046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.292014    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9f7f443-7fdf8647dc-bx9kd"]
Nov 03 21:00:32.393664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.393579    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 21:00:32.399829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.399807    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9f7f443-6965d6db4d-nlrs5"]
Nov 03 21:00:32.492344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.492315    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0749c837-2981-4dec-be3d-703be6cb0825" path="/var/lib/kubelet/pods/0749c837-2981-4dec-be3d-703be6cb0825/volumes"
Nov 03 21:00:32.492723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:32.492709    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" path="/var/lib/kubelet/pods/53787f3d-5d7c-429c-a104-a5c9e11e14fc/volumes"
Nov 03 21:00:35.772560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.772534    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 21:00:35.843994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.843929    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 21:00:35.844378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844359    5533 cpu_manager.go:414] "containerMap: removing container" podUID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" containerName="payload-processing"
Nov 03 21:00:35.844472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844380    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" containerName="payload-processing"
Nov 03 21:00:35.844472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844397    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.844472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844404    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.844472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844436    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0749c837-2981-4dec-be3d-703be6cb0825" containerName="payload-pre-processing"
Nov 03 21:00:35.844472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844444    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0749c837-2981-4dec-be3d-703be6cb0825" containerName="payload-pre-processing"
Nov 03 21:00:35.844738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844542    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.844738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844558    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0749c837-2981-4dec-be3d-703be6cb0825" containerName="payload-pre-processing"
Nov 03 21:00:35.844738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.844572    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="53787f3d-5d7c-429c-a104-a5c9e11e14fc" containerName="payload-processing"
Nov 03 21:00:35.847465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.847446    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:35.864270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.864250    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:00:35.899854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.899829    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e84357\""
Nov 03 21:00:35.899974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.899881    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e84357-dockercfg-dslwt\""
Nov 03 21:00:35.902659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.902619    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 21:00:35.921237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921218    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921270    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921308    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921336    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921356    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921401    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921431    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921484    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921493    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:35.921678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921561    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\") pod \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\" (UID: \"09d753cb-d856-46ee-a29d-a1b4fd36adf6\") "
Nov 03 21:00:35.921744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921681    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:35.921814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921779    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:35.921859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921789    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:35.921859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921830    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:35.922023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.921986    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwq7\" (UniqueName: \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:35.922072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:35.922188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922157    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:35.922188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922182    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:35.922353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922197    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:35.922353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922212    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:35.922353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.922225    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:35.923652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.923616    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:00:35.923740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.923653    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:35.923740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.923668    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "kube-api-access-nh84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:35.923880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.923796    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token" pod "09d753cb-d856-46ee-a29d-a1b4fd36adf6" (UID: "09d753cb-d856-46ee-a29d-a1b4fd36adf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:35.987479 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.987438    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 21:00:35.987950 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.987930    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.987950 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.987950    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.988126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.988042    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerName="istio-proxy"
Nov 03 21:00:35.991460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:35.991438    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.022756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022725    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwq7\" (UniqueName: \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:36.022857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022828    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:36.022903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022888    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:36.022944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022902    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh84c\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-kube-api-access-nh84c\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:36.022944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022913    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:36.022944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.022921    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d753cb-d856-46ee-a29d-a1b4fd36adf6-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:36.023500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.023483    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:36.045691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.045662    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 21:00:36.080831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.080808    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwq7\" (UniqueName: \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\") pod \"payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:36.094725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.094680    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 21:00:36.124350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.124323    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.124442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.124378    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7t7j\" (UniqueName: \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.159048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.159030    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:36.160773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.160750    5533 generic.go:356] "Generic (PLEG): container finished" podID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" containerID="9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0" exitCode=0
Nov 03 21:00:36.160865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.160803    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"
Nov 03 21:00:36.160919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.160803    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerDied","Data":"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"}
Nov 03 21:00:36.160970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.160935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6" event={"ID":"09d753cb-d856-46ee-a29d-a1b4fd36adf6","Type":"ContainerDied","Data":"4bc67c2e954d1b990619d4b08e268d30316f355de666d424eaf3f3a482346f99"}
Nov 03 21:00:36.160970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.160962    5533 scope.go:122] "RemoveContainer" containerID="9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"
Nov 03 21:00:36.172397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.172372    5533 scope.go:122] "RemoveContainer" containerID="3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"
Nov 03 21:00:36.181695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.181673    5533 scope.go:122] "RemoveContainer" containerID="9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"
Nov 03 21:00:36.181982 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:36.181958    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0\": container with ID starting with 9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0 not found: ID does not exist" containerID="9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"
Nov 03 21:00:36.182044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.181995    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0"} err="failed to get container status \"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0\": rpc error: code = NotFound desc = could not find container \"9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0\": container with ID starting with 9a24823cb2afae103e6ada0aa34629470b0c960922a53a8f8497a96702d865b0 not found: ID does not exist"
Nov 03 21:00:36.182044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.182023    5533 scope.go:122] "RemoveContainer" containerID="3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"
Nov 03 21:00:36.182263 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:36.182248    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d\": container with ID starting with 3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d not found: ID does not exist" containerID="3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"
Nov 03 21:00:36.182307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.182270    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d"} err="failed to get container status \"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d\": rpc error: code = NotFound desc = could not find container \"3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d\": container with ID starting with 3656b3013f37f97f0a34467fc79c8e5c523679bb811f54aadccf587b5b1cdd9d not found: ID does not exist"
Nov 03 21:00:36.226328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.226068    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.226662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.226563    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7t7j\" (UniqueName: \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.227565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.227545    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.284393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.284366    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7t7j\" (UniqueName: \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\") pod \"payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.303966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.303941    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:36.318352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.318333    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 21:00:36.327868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.327843    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b32d5a5-openshift-default-77ddbc76cf-tl4l6"]
Nov 03 21:00:36.492011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.491983    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d753cb-d856-46ee-a29d-a1b4fd36adf6" path="/var/lib/kubelet/pods/09d753cb-d856-46ee-a29d-a1b4fd36adf6/volumes"
Nov 03 21:00:36.527126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.527104    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 21:00:36.529486 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:36.529462    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19c9055_2190_4be0_b961_02d4cf3db07c.slice/crio-e0f6374fb70d8ec7dab04f47569846d1794828b1b08a3cc44344e4c21f42db24 WatchSource:0}: Error finding container e0f6374fb70d8ec7dab04f47569846d1794828b1b08a3cc44344e4c21f42db24: Status 404 returned error can't find the container with id e0f6374fb70d8ec7dab04f47569846d1794828b1b08a3cc44344e4c21f42db24
Nov 03 21:00:36.671190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:36.671169    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 21:00:36.672972 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:36.672951    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502ae4a2_d25a_4106_b927_e6041c4d0dbc.slice/crio-bcd779c6ca8fe8b8f7b9b900a987b043e902cfc8b8ab5970ceb897105a2c79b4 WatchSource:0}: Error finding container bcd779c6ca8fe8b8f7b9b900a987b043e902cfc8b8ab5970ceb897105a2c79b4: Status 404 returned error can't find the container with id bcd779c6ca8fe8b8f7b9b900a987b043e902cfc8b8ab5970ceb897105a2c79b4
Nov 03 21:00:37.170871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.170835    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" event={"ID":"502ae4a2-d25a-4106-b927-e6041c4d0dbc","Type":"ContainerStarted","Data":"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"}
Nov 03 21:00:37.171347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.170903    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" event={"ID":"502ae4a2-d25a-4106-b927-e6041c4d0dbc","Type":"ContainerStarted","Data":"bcd779c6ca8fe8b8f7b9b900a987b043e902cfc8b8ab5970ceb897105a2c79b4"}
Nov 03 21:00:37.171347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.170933    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:37.172762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.172732    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" event={"ID":"f19c9055-2190-4be0-b961-02d4cf3db07c","Type":"ContainerStarted","Data":"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"}
Nov 03 21:00:37.172875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.172787    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" event={"ID":"f19c9055-2190-4be0-b961-02d4cf3db07c","Type":"ContainerStarted","Data":"e0f6374fb70d8ec7dab04f47569846d1794828b1b08a3cc44344e4c21f42db24"}
Nov 03 21:00:37.172875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.172815    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:37.282373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.282317    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" podStartSLOduration=2.282298661 podStartE2EDuration="2.282298661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:37.279758871 +0000 UTC m=+738.911072360" watchObservedRunningTime="2026-11-03 21:00:37.282298661 +0000 UTC m=+738.913612139"
Nov 03 21:00:37.390339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:37.390286    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" podStartSLOduration=2.390273556 podStartE2EDuration="2.390273556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:37.38823892 +0000 UTC m=+739.019552421" watchObservedRunningTime="2026-11-03 21:00:37.390273556 +0000 UTC m=+739.021587206"
Nov 03 21:00:38.110572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:38.110543    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 21:00:38.110736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:38.110607    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 21:00:43.803617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:43.803583    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:00:45.639222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.639187    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 21:00:45.643081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.643062    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.695496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.695262    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-791c32\""
Nov 03 21:00:45.695719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.695702    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-791c32-dockercfg-cdljk\""
Nov 03 21:00:45.697984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.697956    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 21:00:45.792875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.792846    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 21:00:45.796762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.796740    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:45.809053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.809019    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwls\" (UniqueName: \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.809167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.809093    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.851717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.851694    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 21:00:45.909837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.909754    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrs82\" (UniqueName: \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:45.909837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.909816    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwls\" (UniqueName: \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.910072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.909902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.910072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.909945    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:45.910837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.910815    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.976608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.976570    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwls\" (UniqueName: \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\") pod \"payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:45.976814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:45.976790    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:46.011414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.011385    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:46.011531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.011465    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrs82\" (UniqueName: \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:46.012102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.012076    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:46.077862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.077825    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrs82\" (UniqueName: \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\") pod \"payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:46.116078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.116049    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:46.342082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.342059    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 21:00:46.343972 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:46.343945    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf497549a_ea48_45fa_86c9_81fb4e98fd5a.slice/crio-e718caddc393ee6a4a66331912920ee74c6d567d9f6af28783b4a6b4167f1f5f WatchSource:0}: Error finding container e718caddc393ee6a4a66331912920ee74c6d567d9f6af28783b4a6b4167f1f5f: Status 404 returned error can't find the container with id e718caddc393ee6a4a66331912920ee74c6d567d9f6af28783b4a6b4167f1f5f
Nov 03 21:00:46.500510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.500486    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 21:00:46.502408 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:00:46.502382    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b82980_14f3_4dc9_80ac_2d8f4111a153.slice/crio-3f3501435e258cd362994536c377e058bd237ee57af98fe9ea20f6a38cac8c91 WatchSource:0}: Error finding container 3f3501435e258cd362994536c377e058bd237ee57af98fe9ea20f6a38cac8c91: Status 404 returned error can't find the container with id 3f3501435e258cd362994536c377e058bd237ee57af98fe9ea20f6a38cac8c91
Nov 03 21:00:46.955738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.955692    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"]
Nov 03 21:00:46.997415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:46.997387    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9f7f443-5m45x"]
Nov 03 21:00:47.231434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.231324    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" event={"ID":"67b82980-14f3-4dc9-80ac-2d8f4111a153","Type":"ContainerStarted","Data":"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"}
Nov 03 21:00:47.231434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.231395    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" event={"ID":"67b82980-14f3-4dc9-80ac-2d8f4111a153","Type":"ContainerStarted","Data":"3f3501435e258cd362994536c377e058bd237ee57af98fe9ea20f6a38cac8c91"}
Nov 03 21:00:47.232040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.231457    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:47.234465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.233940    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" event={"ID":"f497549a-ea48-45fa-86c9-81fb4e98fd5a","Type":"ContainerStarted","Data":"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"}
Nov 03 21:00:47.234465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.233997    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" event={"ID":"f497549a-ea48-45fa-86c9-81fb4e98fd5a","Type":"ContainerStarted","Data":"e718caddc393ee6a4a66331912920ee74c6d567d9f6af28783b4a6b4167f1f5f"}
Nov 03 21:00:47.234465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.234027    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:47.344563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:47.344510    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" podStartSLOduration=2.3444956980000002 podStartE2EDuration="2.344495698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:47.34248506 +0000 UTC m=+748.973798537" watchObservedRunningTime="2026-11-03 21:00:47.344495698 +0000 UTC m=+748.975809176"
Nov 03 21:00:48.182204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.182174    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 21:00:48.182573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.182243    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 21:00:48.292670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.292436    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" podStartSLOduration=3.292416441 podStartE2EDuration="3.292416441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:47.458384868 +0000 UTC m=+749.089698346" watchObservedRunningTime="2026-11-03 21:00:48.292416441 +0000 UTC m=+749.923729929"
Nov 03 21:00:48.495689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.495594    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0" path="/var/lib/kubelet/pods/a14d46e3-a86d-4d8b-8be6-f5d4e4d484f0/volumes"
Nov 03 21:00:48.496256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.496234    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:00:48.502934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.502911    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.593573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.593542    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:00:48.622964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.622939    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\""
Nov 03 21:00:48.636368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636342    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636391    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636411    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636478    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78p6\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636628    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636694    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636746    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636775    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.636839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.636823    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.739761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.739719    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740033    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740280    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740388    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740488    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78p6\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740626    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740797    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.740902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740890    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.741001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740989    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.741359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.741327    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.741509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.740667    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.741627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.741606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.741975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.741957    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.742088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.742067    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.744166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.744143    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.746703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.745717    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.801500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.801470    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.802229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.802207    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78p6\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:48.816433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:48.816412    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:49.280916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:49.280260    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:00:49.283467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:49.283434    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:49.283583 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:49.283516    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:49.283583 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:49.283559    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:49.288290 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:00:49.288262    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:00:50.279056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:50.279018    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerStarted","Data":"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"}
Nov 03 21:00:50.279209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:50.279076    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerStarted","Data":"7c72b1cf0d187c255ecfe4698072a5559191bfcd919f4ac08bfff6928fb6cc1c"}
Nov 03 21:00:50.440415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:50.440351    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podStartSLOduration=2.44033643 podStartE2EDuration="2.44033643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:50.439290917 +0000 UTC m=+752.070604397" watchObservedRunningTime="2026-11-03 21:00:50.44033643 +0000 UTC m=+752.071649907"
Nov 03 21:00:50.817284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:50.817204    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:50.822515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:50.822491    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:51.286034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:51.285998    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:51.287306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:51.287271    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:00:54.305736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.305705    5533 generic.go:356] "Generic (PLEG): container finished" podID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerID="e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567" exitCode=0
Nov 03 21:00:54.306154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.305781    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerDied","Data":"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"}
Nov 03 21:00:54.306154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.306128    5533 scope.go:122] "RemoveContainer" containerID="e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"
Nov 03 21:00:54.307316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.307280    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:54.307438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.307369    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:54.307438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:54.307417    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:00:55.313546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:55.313505    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerStarted","Data":"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"}
Nov 03 21:00:55.774666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:55.773987    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:00:55.774666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:55.774101    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:00:55.774666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:55.774141    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:00:56.775969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:56.775934    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:00:56.776391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:56.775989    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:00:57.773656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:57.773133    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:00:57.773656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:57.773213    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:00:58.246271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:58.246241    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 21:00:58.246660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:58.246427    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 21:00:58.774660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:58.774115    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:00:58.774660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:58.774183    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:00:59.774659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:59.773961    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:00:59.774659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:00:59.774018    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:01:00.221832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:00.221798    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:01:00.775884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:00.775849    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:01:00.776276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:00.776143    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:01:00.776882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:00.776867    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 21:01:17.328873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:17.328841    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:01:19.259597 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:01:19.259558    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:01:25.972294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:25.971828    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 21:01:25.980758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:25.980731    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.030056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.030024    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 21:01:26.082003 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.081970    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"kube-root-ca.crt\""
Nov 03 21:01:26.082220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.082056    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"default-dockercfg-dksr6\""
Nov 03 21:01:26.082220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.082155    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"simulator-w2-099a5e-kserve-self-signed-certs\""
Nov 03 21:01:26.082220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.082219    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"openshift-service-ca.crt\""
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165153    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165251    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165279    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165320    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn85m\" (UniqueName: \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165362    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.167662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.165407    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271147    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271256    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271340    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271369    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271407    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn85m\" (UniqueName: \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.272720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.271447    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.275655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.273993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.275655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.274270    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.275655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.274858    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.275655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.274887    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.277187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.277159    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.332429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.332405    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn85m\" (UniqueName: \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\") pod \"simulator-w2-099a5e-kserve-646b68d7c6-9777h\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") " pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.593066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.592990    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:26.981661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:26.980343    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 21:01:27.504192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:27.504150    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" event={"ID":"45b1b737-1590-48aa-b291-5a2b465c779c","Type":"ContainerStarted","Data":"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"}
Nov 03 21:01:27.504382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:27.504202    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" event={"ID":"45b1b737-1590-48aa-b291-5a2b465c779c","Type":"ContainerStarted","Data":"fce1358bdf345cd824970ab426cc41214eaf7febd08538437b0c47505211f587"}
Nov 03 21:01:27.593815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:27.593782    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:27.617617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:27.617565    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podStartSLOduration=2.6175494820000003 podStartE2EDuration="2.617549482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:27.616341931 +0000 UTC m=+789.247655422" watchObservedRunningTime="2026-11-03 21:01:27.617549482 +0000 UTC m=+789.248862960"
Nov 03 21:01:34.546763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.546705    5533 generic.go:356] "Generic (PLEG): container finished" podID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerID="0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967" exitCode=0
Nov 03 21:01:34.547114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.546776    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerDied","Data":"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"}
Nov 03 21:01:34.547114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.547102    5533 scope.go:122] "RemoveContainer" containerID="0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"
Nov 03 21:01:34.548285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.548257    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:34.548343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.548330    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:34.548379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:34.548368    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:01:35.574317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:35.573392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerStarted","Data":"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"}
Nov 03 21:01:35.817052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:35.817021    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:01:35.818211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:35.818187    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:01:35.818333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:35.818236    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:01:36.286427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.286389    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:01:36.593605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.593524    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:36.607243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.607219    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:36.817557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.817519    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:01:36.817735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.817587    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:01:36.995467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:36.995429    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:01:37.611167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:37.611124    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:01:37.817606 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:37.817573    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:01:37.817772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:37.817661    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:01:38.817715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:38.817680    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:01:38.818108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:38.817743    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:01:39.817389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:39.817351    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:01:39.817563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:39.817413    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:01:40.820450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:40.820421    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:01:40.821669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:40.821646    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:01:49.261466 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:01:49.261431    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:01:52.426607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.426572    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 21:01:52.430419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.430397    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.525455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.525421    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.525612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.525497    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.526005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.525971    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.526232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526168    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.526360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526309    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2xz\" (UniqueName: \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.526408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526377    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.526560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526543    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"kube-root-ca.crt\""
Nov 03 21:01:52.526669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526627    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"openshift-service-ca.crt\""
Nov 03 21:01:52.526794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526776    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"premium-w3-320f60-kserve-self-signed-certs\""
Nov 03 21:01:52.526897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.526791    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"default-dockercfg-mhpg2\""
Nov 03 21:01:52.528825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.528808    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 21:01:52.627259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627224    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627264    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627342    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627367    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627388    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2xz\" (UniqueName: \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627408    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627749    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627778    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.627892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.627856    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.629554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.629527    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.629833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.629816    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.712654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.712616    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2xz\" (UniqueName: \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\") pod \"premium-w3-320f60-kserve-5d8656877b-nv7rj\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") " pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:52.741425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:52.741401    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:53.107349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:53.107321    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 21:01:53.109748 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:01:53.109715    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbc95a86_3a17_4287_afad_4b0dbdf44282.slice/crio-1855a767ad0f1a6bcd9b5514652886c3958a609f231c9b77ee5edd7b7fbcb2de WatchSource:0}: Error finding container 1855a767ad0f1a6bcd9b5514652886c3958a609f231c9b77ee5edd7b7fbcb2de: Status 404 returned error can't find the container with id 1855a767ad0f1a6bcd9b5514652886c3958a609f231c9b77ee5edd7b7fbcb2de
Nov 03 21:01:53.692327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:53.692293    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" event={"ID":"cbc95a86-3a17-4287-afad-4b0dbdf44282","Type":"ContainerStarted","Data":"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"}
Nov 03 21:01:53.692698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:53.692341    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" event={"ID":"cbc95a86-3a17-4287-afad-4b0dbdf44282","Type":"ContainerStarted","Data":"1855a767ad0f1a6bcd9b5514652886c3958a609f231c9b77ee5edd7b7fbcb2de"}
Nov 03 21:01:53.741499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:53.741469    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:01:53.802232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:53.802179    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" podStartSLOduration=1.802162668 podStartE2EDuration="1.802162668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:53.801467208 +0000 UTC m=+815.432780686" watchObservedRunningTime="2026-11-03 21:01:53.802162668 +0000 UTC m=+815.433476147"
Nov 03 21:01:59.880305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:01:59.880225    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:02:02.741582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:02.741547    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:02:02.754051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:02.754026    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:02:03.753609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:03.753581    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 21:02:19.263533 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:02:19.263491    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:02:19.825104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:19.825062    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:02:29.806745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.806714    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 21:02:29.810541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.810521    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.861115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.861090    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"default-dockercfg-q4tcx\""
Nov 03 21:02:29.864608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864573    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.864740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864630    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.864789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864748    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.864831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864786    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.864897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864881    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.864983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.864962    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjkq\" (UniqueName: \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.908031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.907998    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"premium-w4-e84357-kserve-self-signed-certs\""
Nov 03 21:02:29.908031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.908021    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"openshift-service-ca.crt\""
Nov 03 21:02:29.912209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.912185    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 21:02:29.914171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.914154    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e84357\"/\"kube-root-ca.crt\""
Nov 03 21:02:29.966304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966272    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.966474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966335    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjkq\" (UniqueName: \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.966474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966398    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.966474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966433    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.966474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966460    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.966773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.966486    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.967088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.967054    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.967192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.967169    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.967755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.967733    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.970324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.970302    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:29.971127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:29.971106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:30.026136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:30.026106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjkq\" (UniqueName: \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\") pod \"premium-w4-e84357-kserve-54d766fdcd-24tr2\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") " pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:30.122988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:30.122902    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:30.497023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:30.496998    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 21:02:30.498854 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:02:30.498823    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95fad564_be5b_462d_a0a3_3a48c7826f63.slice/crio-4e271a6d75e42654f89c3a6e2ff96a97d4014353546b9e1c228dcd79c33bda46 WatchSource:0}: Error finding container 4e271a6d75e42654f89c3a6e2ff96a97d4014353546b9e1c228dcd79c33bda46: Status 404 returned error can't find the container with id 4e271a6d75e42654f89c3a6e2ff96a97d4014353546b9e1c228dcd79c33bda46
Nov 03 21:02:30.909085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:30.909001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" event={"ID":"95fad564-be5b-462d-a0a3-3a48c7826f63","Type":"ContainerStarted","Data":"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"}
Nov 03 21:02:30.909085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:30.909046    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" event={"ID":"95fad564-be5b-462d-a0a3-3a48c7826f63","Type":"ContainerStarted","Data":"4e271a6d75e42654f89c3a6e2ff96a97d4014353546b9e1c228dcd79c33bda46"}
Nov 03 21:02:31.019902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:31.019856    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podStartSLOduration=2.019844099 podStartE2EDuration="2.019844099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:31.01841368 +0000 UTC m=+852.649727157" watchObservedRunningTime="2026-11-03 21:02:31.019844099 +0000 UTC m=+852.651157576"
Nov 03 21:02:31.124144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:31.124110    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:37.408952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:37.408917    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:02:40.123988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:40.123939    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:40.136295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:40.136273    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:40.978004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:02:40.977962    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:02:49.260196 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:02:49.260159    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:03:03.736504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:03.736465    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:03:08.213666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.213623    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 21:03:08.217228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.217211    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.313919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.313887    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"kube-root-ca.crt\""
Nov 03 21:03:08.314070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.313925    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"openshift-service-ca.crt\""
Nov 03 21:03:08.314070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.313963    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"premium-w0-791c32-kserve-self-signed-certs\""
Nov 03 21:03:08.314070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.314035    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-791c32\"/\"default-dockercfg-t5gcp\""
Nov 03 21:03:08.316127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.316108    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 21:03:08.318481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318461    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxmp\" (UniqueName: \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.318526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318496    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.318576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318524    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.318701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318684    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.318773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318749    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.318831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.318797    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419566    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxmp\" (UniqueName: \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419609    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419656    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419696    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419752    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.419874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.419796    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.420196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.420170    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.420265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.420235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.420402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.420385    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.422054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.422028    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.422355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.422335    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.478879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.478823    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxmp\" (UniqueName: \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\") pod \"premium-w0-791c32-kserve-dbb6d88d5-tllrq\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") " pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.529014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.528985    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:08.897857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:08.897830    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 21:03:08.899387 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:03:08.899361    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a72c370_7a7f_4c86_a56f_8a662d457a80.slice/crio-0d1708ea2a753cdf6c7be4774f23860cb7e430555a4cc005799b8026b2260c31 WatchSource:0}: Error finding container 0d1708ea2a753cdf6c7be4774f23860cb7e430555a4cc005799b8026b2260c31: Status 404 returned error can't find the container with id 0d1708ea2a753cdf6c7be4774f23860cb7e430555a4cc005799b8026b2260c31
Nov 03 21:03:09.121987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:09.121956    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" event={"ID":"0a72c370-7a7f-4c86-a56f-8a662d457a80","Type":"ContainerStarted","Data":"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"}
Nov 03 21:03:09.122215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:09.121996    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" event={"ID":"0a72c370-7a7f-4c86-a56f-8a662d457a80","Type":"ContainerStarted","Data":"0d1708ea2a753cdf6c7be4774f23860cb7e430555a4cc005799b8026b2260c31"}
Nov 03 21:03:09.235043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:09.234994    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podStartSLOduration=1.234982778 podStartE2EDuration="1.234982778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:09.233386995 +0000 UTC m=+890.864700472" watchObservedRunningTime="2026-11-03 21:03:09.234982778 +0000 UTC m=+890.866296251"
Nov 03 21:03:09.529462 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:09.529388    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:14.244003 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.243972    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"]
Nov 03 21:03:14.248014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.247989    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.346940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.346904    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:14.346940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.346914    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:03:14.349140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.349111    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"]
Nov 03 21:03:14.369162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.369131    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.369260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.369187    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.369393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.369375    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8m6\" (UniqueName: \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.470261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.470227    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.470371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.470292    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.470417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.470390    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8m6\" (UniqueName: \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.470898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.470865    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.472594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.472576    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.530198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.530141    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8m6\" (UniqueName: \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\") pod \"maas-api-revoke-sub-79733355-gschr\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.560986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.560964    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:14.927189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:14.927156    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"]
Nov 03 21:03:14.928517 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:03:14.928481    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18cfa04_f452_44c2_96c2_c4e2d4ddf8a9.slice/crio-cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8 WatchSource:0}: Error finding container cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8: Status 404 returned error can't find the container with id cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8
Nov 03 21:03:15.165243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:15.165213    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr" event={"ID":"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9","Type":"ContainerStarted","Data":"cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8"}
Nov 03 21:03:16.173904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:16.173862    5533 generic.go:356] "Generic (PLEG): container finished" podID="b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" containerID="601ceb1a73959e05b135a3c9246ea525227d9cb439c028aeb3e4f5cc5f4a25e7" exitCode=0
Nov 03 21:03:16.174307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:16.173937    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr" event={"ID":"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9","Type":"ContainerDied","Data":"601ceb1a73959e05b135a3c9246ea525227d9cb439c028aeb3e4f5cc5f4a25e7"}
Nov 03 21:03:17.314891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.314869    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:17.395756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.395732    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\" (UniqueName: \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\") pod \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") "
Nov 03 21:03:17.395863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.395781    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\") pod \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") "
Nov 03 21:03:17.395863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.395822    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\") pod \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\" (UID: \"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9\") "
Nov 03 21:03:17.396139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.396107    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca" pod "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" (UID: "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:17.397775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.397750    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca" pod "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" (UID: "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:17.397889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.397870    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6" pod "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" (UID: "b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9"). InnerVolumeSpecName "kube-api-access-qd8m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:17.497139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.497062    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd8m6\" (UniqueName: \"kubernetes.io/projected/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-kube-api-access-qd8m6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:17.497139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.497089    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:17.497139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:17.497099    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:18.186945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.186911    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr" event={"ID":"b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9","Type":"ContainerDied","Data":"cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8"}
Nov 03 21:03:18.187115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.186948    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"
Nov 03 21:03:18.187115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.186959    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cea4a15b14f3975168d5b196e2dea4909c274bcd6a4facfb3029895cf6913fb8"
Nov 03 21:03:18.529773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.529695    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:18.542107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.542081    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:18.585776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.585746    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:03:18.588663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.588615    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:03:18.606230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.606205    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:03:18.607249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:18.607223    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:03:19.204487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:19.204460    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 21:03:19.260683 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:03:19.260648    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:03:32.090394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.090359    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"]
Nov 03 21:03:32.090983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.090962    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" containerName="revoke-keys"
Nov 03 21:03:32.091078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.090986    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" containerName="revoke-keys"
Nov 03 21:03:32.091155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.091142    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" containerName="revoke-keys"
Nov 03 21:03:32.094612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.094590    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.126881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.126856    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49475\" (UniqueName: \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.126993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.126916    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.127057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.127006    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.145853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.145828    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:32.149035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.149006    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"]
Nov 03 21:03:32.192561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.192542    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:03:32.228223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.228197    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49475\" (UniqueName: \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.228307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.228261    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.228347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.228309    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.228948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.228926    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.230624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.230605    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.285792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.285767    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49475\" (UniqueName: \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\") pod \"maas-api-revoke-sub-9ada009b-6xcbw\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.408587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.408523    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:32.777725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:32.777693    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"]
Nov 03 21:03:32.781491 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:03:32.781463    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddca59f1_e996_466a_8110_c7c113ff9ad7.slice/crio-7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca WatchSource:0}: Error finding container 7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca: Status 404 returned error can't find the container with id 7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca
Nov 03 21:03:33.283372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:33.283334    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw" event={"ID":"ddca59f1-e996-466a-8110-c7c113ff9ad7","Type":"ContainerStarted","Data":"aab1f42505e9f7202680c3dc774357ce7c7514fe5845817bdfa9e60044ab5cec"}
Nov 03 21:03:33.283786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:33.283395    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw" event={"ID":"ddca59f1-e996-466a-8110-c7c113ff9ad7","Type":"ContainerStarted","Data":"7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca"}
Nov 03 21:03:33.393682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:33.393564    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw" podStartSLOduration=1.393545676 podStartE2EDuration="1.393545676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:33.391330972 +0000 UTC m=+915.022644447" watchObservedRunningTime="2026-11-03 21:03:33.393545676 +0000 UTC m=+915.024859156"
Nov 03 21:03:34.253231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:34.253194    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:03:34.290118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:34.290087    5533 generic.go:356] "Generic (PLEG): container finished" podID="ddca59f1-e996-466a-8110-c7c113ff9ad7" containerID="aab1f42505e9f7202680c3dc774357ce7c7514fe5845817bdfa9e60044ab5cec" exitCode=0
Nov 03 21:03:34.290466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:34.290163    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw" event={"ID":"ddca59f1-e996-466a-8110-c7c113ff9ad7","Type":"ContainerDied","Data":"aab1f42505e9f7202680c3dc774357ce7c7514fe5845817bdfa9e60044ab5cec"}
Nov 03 21:03:35.428718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.428695    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:35.561799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.561743    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\" (UniqueName: \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\") pod \"ddca59f1-e996-466a-8110-c7c113ff9ad7\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") "
Nov 03 21:03:35.561799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.561778    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\") pod \"ddca59f1-e996-466a-8110-c7c113ff9ad7\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") "
Nov 03 21:03:35.561997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.561850    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\") pod \"ddca59f1-e996-466a-8110-c7c113ff9ad7\" (UID: \"ddca59f1-e996-466a-8110-c7c113ff9ad7\") "
Nov 03 21:03:35.563860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.563826    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca" pod "ddca59f1-e996-466a-8110-c7c113ff9ad7" (UID: "ddca59f1-e996-466a-8110-c7c113ff9ad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:35.564286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.564261    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475" pod "ddca59f1-e996-466a-8110-c7c113ff9ad7" (UID: "ddca59f1-e996-466a-8110-c7c113ff9ad7"). InnerVolumeSpecName "kube-api-access-49475". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:35.564469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.564449    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca" pod "ddca59f1-e996-466a-8110-c7c113ff9ad7" (UID: "ddca59f1-e996-466a-8110-c7c113ff9ad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:35.663601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.663571    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.663601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.663594    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49475\" (UniqueName: \"kubernetes.io/projected/ddca59f1-e996-466a-8110-c7c113ff9ad7-kube-api-access-49475\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.663601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:35.663605    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddca59f1-e996-466a-8110-c7c113ff9ad7-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:36.301960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:36.301932    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"
Nov 03 21:03:36.302141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:36.301937    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw" event={"ID":"ddca59f1-e996-466a-8110-c7c113ff9ad7","Type":"ContainerDied","Data":"7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca"}
Nov 03 21:03:36.302141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:36.302060    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cee9d740e04abf777e1c4b491ce0b0ff3e7f0ea51f2ae8dca2c898c93da01ca"
Nov 03 21:03:43.550714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:43.550679    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 21:03:43.551255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:43.550868    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy" containerID="cri-o://9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434" gracePeriod=30
Nov 03 21:03:45.948802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:45.948767    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"]
Nov 03 21:03:45.949201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:45.949185    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ddca59f1-e996-466a-8110-c7c113ff9ad7" containerName="revoke-keys"
Nov 03 21:03:45.949201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:45.949197    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddca59f1-e996-466a-8110-c7c113ff9ad7" containerName="revoke-keys"
Nov 03 21:03:45.949280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:45.949272    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddca59f1-e996-466a-8110-c7c113ff9ad7" containerName="revoke-keys"
Nov 03 21:03:45.952761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:45.952740    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.050679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.050628    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:46.050904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.050881    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:03:46.053084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.053060    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"]
Nov 03 21:03:46.053353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.053336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.053395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.053382    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwjl\" (UniqueName: \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.053569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.053554    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.154827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.154798    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.154980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.154851    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.154980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.154888    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwjl\" (UniqueName: \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.155581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.155562    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.157209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.157175    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.212464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.212404    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwjl\" (UniqueName: \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\") pod \"maas-api-revoke-sub-4523182d-zk4r5\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.265126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.265101    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:46.634197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:46.634172    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"]
Nov 03 21:03:46.636480 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:03:46.636453    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7636c2a5_50a7_484a_b73d_ce006787a3ff.slice/crio-647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2 WatchSource:0}: Error finding container 647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2: Status 404 returned error can't find the container with id 647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2
Nov 03 21:03:47.365342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:47.365306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5" event={"ID":"7636c2a5-50a7-484a-b73d-ce006787a3ff","Type":"ContainerStarted","Data":"4acf8c2134efecdbe32180d480d0b6a3ef66a084e541d61b31e6224be558c0ea"}
Nov 03 21:03:47.365751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:47.365366    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5" event={"ID":"7636c2a5-50a7-484a-b73d-ce006787a3ff","Type":"ContainerStarted","Data":"647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2"}
Nov 03 21:03:47.478004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:47.477955    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5" podStartSLOduration=2.477940953 podStartE2EDuration="2.477940953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:47.475135621 +0000 UTC m=+929.106449098" watchObservedRunningTime="2026-11-03 21:03:47.477940953 +0000 UTC m=+929.109254454"
Nov 03 21:03:48.072648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.072612    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"]
Nov 03 21:03:48.076198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.076174    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.175603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.175565    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"]
Nov 03 21:03:48.177214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.177188    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.177391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.177228    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxh7j\" (UniqueName: \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.177460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.177406    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.278248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.278216    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.278399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.278273    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxh7j\" (UniqueName: \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.278461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.278395    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.278946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.278919    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.280800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.280778    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.336075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.336018    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxh7j\" (UniqueName: \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\") pod \"maas-api-revoke-sub-3b0fc0f3-htz69\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.374191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.374160    5533 generic.go:356] "Generic (PLEG): container finished" podID="7636c2a5-50a7-484a-b73d-ce006787a3ff" containerID="4acf8c2134efecdbe32180d480d0b6a3ef66a084e541d61b31e6224be558c0ea" exitCode=0
Nov 03 21:03:48.374509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.374212    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5" event={"ID":"7636c2a5-50a7-484a-b73d-ce006787a3ff","Type":"ContainerDied","Data":"4acf8c2134efecdbe32180d480d0b6a3ef66a084e541d61b31e6224be558c0ea"}
Nov 03 21:03:48.392442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.392421    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:48.764133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.764100    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"]
Nov 03 21:03:48.808858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.808832    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 21:03:48.884285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884260    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884336    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884381    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884417    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884611    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884678    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884769    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884811    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.884857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.884848    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\") pod \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\" (UID: \"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3\") "
Nov 03 21:03:48.885866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885070    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:48.885866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885250    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.885866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885294    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:48.885866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885595    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:48.885866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885781    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:48.886122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.885399    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:48.887045 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.887016    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "kube-api-access-m7vwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:48.887150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.887100    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:48.887574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.887550    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:48.887740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.887705    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy" pod "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" (UID: "6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:48.986704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986681    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986705    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986714    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986723    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986733    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986746    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vwl\" (UniqueName: \"kubernetes.io/projected/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-kube-api-access-m7vwl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986755    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:48.986841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:48.986765    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:49.262002 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:03:49.261911    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:03:49.382440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.382401    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerID="9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434" exitCode=0
Nov 03 21:03:49.382904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.382453    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"
Nov 03 21:03:49.382904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.382485    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerDied","Data":"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"}
Nov 03 21:03:49.382904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.382533    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh" event={"ID":"6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3","Type":"ContainerDied","Data":"96c097d5d6b4d9ba70ac3712d29340a66f0b0876fee5afaa39aea028f139bd4e"}
Nov 03 21:03:49.382904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.382564    5533 scope.go:122] "RemoveContainer" containerID="9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"
Nov 03 21:03:49.384525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.384499    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69" event={"ID":"d07b29e4-29cb-48b6-9db7-a0662294c6e6","Type":"ContainerStarted","Data":"d76c73a735b430bc0bd167a8eb8456f58e3addf4b4e2db2054d0586766aa7f78"}
Nov 03 21:03:49.384735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.384562    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69" event={"ID":"d07b29e4-29cb-48b6-9db7-a0662294c6e6","Type":"ContainerStarted","Data":"3c3bec65ca97bbbfd3391f9c7f953289bfc4fb3ec06879a8fb9d9989bcc88ace"}
Nov 03 21:03:49.393205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.393187    5533 scope.go:122] "RemoveContainer" containerID="1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"
Nov 03 21:03:49.402576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.402427    5533 scope.go:122] "RemoveContainer" containerID="9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"
Nov 03 21:03:49.402764 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:03:49.402732    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434\": container with ID starting with 9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434 not found: ID does not exist" containerID="9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"
Nov 03 21:03:49.402876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.402774    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434"} err="failed to get container status \"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434\": rpc error: code = NotFound desc = could not find container \"9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434\": container with ID starting with 9725be7b4389031a3d05d42541181a17e6c396a854fd8f132395794ac0272434 not found: ID does not exist"
Nov 03 21:03:49.402876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.402798    5533 scope.go:122] "RemoveContainer" containerID="1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"
Nov 03 21:03:49.403106 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:03:49.403088    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e\": container with ID starting with 1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e not found: ID does not exist" containerID="1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"
Nov 03 21:03:49.403157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.403113    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e"} err="failed to get container status \"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e\": rpc error: code = NotFound desc = could not find container \"1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e\": container with ID starting with 1a847faff82706d09300cc3e2a02cebfb7349f6a631afe0c27819f97e430b15e not found: ID does not exist"
Nov 03 21:03:49.526526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.526472    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:49.543662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.543605    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69" podStartSLOduration=1.543593015 podStartE2EDuration="1.543593015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:49.541948082 +0000 UTC m=+931.173261561" watchObservedRunningTime="2026-11-03 21:03:49.543593015 +0000 UTC m=+931.174906491"
Nov 03 21:03:49.648041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.648015    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 21:03:49.693885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.693861    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\" (UniqueName: \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\") pod \"7636c2a5-50a7-484a-b73d-ce006787a3ff\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") "
Nov 03 21:03:49.693997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.693943    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\") pod \"7636c2a5-50a7-484a-b73d-ce006787a3ff\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") "
Nov 03 21:03:49.693997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.693984    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\") pod \"7636c2a5-50a7-484a-b73d-ce006787a3ff\" (UID: \"7636c2a5-50a7-484a-b73d-ce006787a3ff\") "
Nov 03 21:03:49.694287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.694265    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca" pod "7636c2a5-50a7-484a-b73d-ce006787a3ff" (UID: "7636c2a5-50a7-484a-b73d-ce006787a3ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:49.694831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.694810    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-hb5wh"]
Nov 03 21:03:49.695927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.695901    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl" pod "7636c2a5-50a7-484a-b73d-ce006787a3ff" (UID: "7636c2a5-50a7-484a-b73d-ce006787a3ff"). InnerVolumeSpecName "kube-api-access-lwwjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:49.696009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.695994    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca" pod "7636c2a5-50a7-484a-b73d-ce006787a3ff" (UID: "7636c2a5-50a7-484a-b73d-ce006787a3ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:49.795682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.795593    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:49.795682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.795619    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7636c2a5-50a7-484a-b73d-ce006787a3ff-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:49.795682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:49.795658    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwjl\" (UniqueName: \"kubernetes.io/projected/7636c2a5-50a7-484a-b73d-ce006787a3ff-kube-api-access-lwwjl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.397660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.396876    5533 generic.go:356] "Generic (PLEG): container finished" podID="d07b29e4-29cb-48b6-9db7-a0662294c6e6" containerID="d76c73a735b430bc0bd167a8eb8456f58e3addf4b4e2db2054d0586766aa7f78" exitCode=0
Nov 03 21:03:50.397660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.397000    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69" event={"ID":"d07b29e4-29cb-48b6-9db7-a0662294c6e6","Type":"ContainerDied","Data":"d76c73a735b430bc0bd167a8eb8456f58e3addf4b4e2db2054d0586766aa7f78"}
Nov 03 21:03:50.405691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.398465    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5" event={"ID":"7636c2a5-50a7-484a-b73d-ce006787a3ff","Type":"ContainerDied","Data":"647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2"}
Nov 03 21:03:50.405691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.398501    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647d65753e2068a45bd8e93ee0e5ffe4f9e5122bccadbc50116d1fb4cc464ec2"
Nov 03 21:03:50.405691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.398514    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"
Nov 03 21:03:50.494941 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:50.494908    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" path="/var/lib/kubelet/pods/6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3/volumes"
Nov 03 21:03:51.540529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.540505    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:51.713809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.713782    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\") pod \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") "
Nov 03 21:03:51.714107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.713843    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\" (UniqueName: \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\") pod \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") "
Nov 03 21:03:51.714107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.713861    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\") pod \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\" (UID: \"d07b29e4-29cb-48b6-9db7-a0662294c6e6\") "
Nov 03 21:03:51.714299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.714201    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca" pod "d07b29e4-29cb-48b6-9db7-a0662294c6e6" (UID: "d07b29e4-29cb-48b6-9db7-a0662294c6e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:51.716022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.715995    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca" pod "d07b29e4-29cb-48b6-9db7-a0662294c6e6" (UID: "d07b29e4-29cb-48b6-9db7-a0662294c6e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:51.716153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.716082    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j" pod "d07b29e4-29cb-48b6-9db7-a0662294c6e6" (UID: "d07b29e4-29cb-48b6-9db7-a0662294c6e6"). InnerVolumeSpecName "kube-api-access-gxh7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:51.815193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.815164    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:51.815193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.815192    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxh7j\" (UniqueName: \"kubernetes.io/projected/d07b29e4-29cb-48b6-9db7-a0662294c6e6-kube-api-access-gxh7j\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:51.815366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:51.815203    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07b29e4-29cb-48b6-9db7-a0662294c6e6-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:52.411194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:52.411151    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69" event={"ID":"d07b29e4-29cb-48b6-9db7-a0662294c6e6","Type":"ContainerDied","Data":"3c3bec65ca97bbbfd3391f9c7f953289bfc4fb3ec06879a8fb9d9989bcc88ace"}
Nov 03 21:03:52.411361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:52.411172    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"
Nov 03 21:03:52.411361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:52.411248    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c3bec65ca97bbbfd3391f9c7f953289bfc4fb3ec06879a8fb9d9989bcc88ace"
Nov 03 21:03:53.050589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.050556    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"]
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.050965    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7636c2a5-50a7-484a-b73d-ce006787a3ff" containerName="revoke-keys"
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.050977    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7636c2a5-50a7-484a-b73d-ce006787a3ff" containerName="revoke-keys"
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.050989    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.050995    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.051022    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d07b29e4-29cb-48b6-9db7-a0662294c6e6" containerName="revoke-keys"
Nov 03 21:03:53.051077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.051027    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07b29e4-29cb-48b6-9db7-a0662294c6e6" containerName="revoke-keys"
Nov 03 21:03:53.051400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.051087    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7636c2a5-50a7-484a-b73d-ce006787a3ff" containerName="revoke-keys"
Nov 03 21:03:53.051400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.051097    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:03:53.051400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.051107    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07b29e4-29cb-48b6-9db7-a0662294c6e6" containerName="revoke-keys"
Nov 03 21:03:53.054752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.054727    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.108318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.108293    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"]
Nov 03 21:03:53.152394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.152374    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:53.152536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.152462    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:03:53.228971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.228949    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.229073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.228981    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chf7s\" (UniqueName: \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.229139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.229123    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.330218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.330159    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.330375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.330253    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.330375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.330286    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chf7s\" (UniqueName: \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.330870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.330847    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.332540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.332518    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.388362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.388330    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chf7s\" (UniqueName: \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\") pod \"maas-api-revoke-sub-06fb7af4-gr97v\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:53.668851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:53.668781    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:54.033226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:54.033198    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"]
Nov 03 21:03:54.035024 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:03:54.035000    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be50500_cbda_4852_95ad_79a09fecd980.slice/crio-549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70 WatchSource:0}: Error finding container 549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70: Status 404 returned error can't find the container with id 549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70
Nov 03 21:03:54.426930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:54.426898    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v" event={"ID":"7be50500-cbda-4852-95ad-79a09fecd980","Type":"ContainerStarted","Data":"473eba31932c35733452b253e3685420e3fab87240af5d4d8c01cb5d157266d0"}
Nov 03 21:03:54.427296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:54.426952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v" event={"ID":"7be50500-cbda-4852-95ad-79a09fecd980","Type":"ContainerStarted","Data":"549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70"}
Nov 03 21:03:54.537951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:54.537906    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v" podStartSLOduration=1.537892683 podStartE2EDuration="1.537892683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:54.536004724 +0000 UTC m=+936.167318201" watchObservedRunningTime="2026-11-03 21:03:54.537892683 +0000 UTC m=+936.169206160"
Nov 03 21:03:55.436601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:55.436563    5533 generic.go:356] "Generic (PLEG): container finished" podID="7be50500-cbda-4852-95ad-79a09fecd980" containerID="473eba31932c35733452b253e3685420e3fab87240af5d4d8c01cb5d157266d0" exitCode=0
Nov 03 21:03:55.437015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:55.436609    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v" event={"ID":"7be50500-cbda-4852-95ad-79a09fecd980","Type":"ContainerDied","Data":"473eba31932c35733452b253e3685420e3fab87240af5d4d8c01cb5d157266d0"}
Nov 03 21:03:56.587882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.587860    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:56.765221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.765156    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\") pod \"7be50500-cbda-4852-95ad-79a09fecd980\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") "
Nov 03 21:03:56.765358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.765262    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\") pod \"7be50500-cbda-4852-95ad-79a09fecd980\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") "
Nov 03 21:03:56.765358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.765323    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\" (UniqueName: \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\") pod \"7be50500-cbda-4852-95ad-79a09fecd980\" (UID: \"7be50500-cbda-4852-95ad-79a09fecd980\") "
Nov 03 21:03:56.765666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.765617    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca" pod "7be50500-cbda-4852-95ad-79a09fecd980" (UID: "7be50500-cbda-4852-95ad-79a09fecd980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:56.767404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.767381    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca" pod "7be50500-cbda-4852-95ad-79a09fecd980" (UID: "7be50500-cbda-4852-95ad-79a09fecd980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:56.767488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.767420    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s" pod "7be50500-cbda-4852-95ad-79a09fecd980" (UID: "7be50500-cbda-4852-95ad-79a09fecd980"). InnerVolumeSpecName "kube-api-access-chf7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:56.866414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.866386    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chf7s\" (UniqueName: \"kubernetes.io/projected/7be50500-cbda-4852-95ad-79a09fecd980-kube-api-access-chf7s\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:56.866414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.866412    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:56.866572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:56.866421    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be50500-cbda-4852-95ad-79a09fecd980-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:57.456203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:57.456173    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"
Nov 03 21:03:57.456371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:57.456214    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v" event={"ID":"7be50500-cbda-4852-95ad-79a09fecd980","Type":"ContainerDied","Data":"549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70"}
Nov 03 21:03:57.456371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:03:57.456258    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549db7e8b563508e19bd2cd16100540c21577c756392e2280cb0463163ac5a70"
Nov 03 21:04:01.398451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.398412    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"]
Nov 03 21:04:01.399103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399079    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7be50500-cbda-4852-95ad-79a09fecd980" containerName="revoke-keys"
Nov 03 21:04:01.399188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399104    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be50500-cbda-4852-95ad-79a09fecd980" containerName="revoke-keys"
Nov 03 21:04:01.399188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399167    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:04:01.399188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399176    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:04:01.399390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399294    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4cbc95-15ee-4a79-a0a0-8887d4bf4bc3" containerName="istio-proxy"
Nov 03 21:04:01.399390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.399310    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be50500-cbda-4852-95ad-79a09fecd980" containerName="revoke-keys"
Nov 03 21:04:01.403116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.403093    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.454765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.454743    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:01.457602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.457578    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"]
Nov 03 21:04:01.501619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.501588    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:04:01.501819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.501800    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnx5\" (UniqueName: \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.501926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.501912    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.501987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.501959    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.602971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.602946    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnx5\" (UniqueName: \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.603085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.602982    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.603085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.603006    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.603737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.603712    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.605224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.605204    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.660259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.660206    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnx5\" (UniqueName: \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\") pod \"maas-api-revoke-sub-c9dcb5a1-mqtdh\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:01.717191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:01.717167    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:02.092436 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:02.092397    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda168e771_2b8c_4c44_8cd7_f040996a98d1.slice/crio-6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494 WatchSource:0}: Error finding container 6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494: Status 404 returned error can't find the container with id 6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494
Nov 03 21:04:02.093105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:02.093040    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"]
Nov 03 21:04:02.484824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:02.484794    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh" event={"ID":"a168e771-2b8c-4c44-8cd7-f040996a98d1","Type":"ContainerStarted","Data":"b4a35b26abe0e8756ba90fcc276c400298b629a7763831393387423cb7042c76"}
Nov 03 21:04:02.485139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:02.484844    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh" event={"ID":"a168e771-2b8c-4c44-8cd7-f040996a98d1","Type":"ContainerStarted","Data":"6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494"}
Nov 03 21:04:02.595762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:02.595708    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh" podStartSLOduration=1.59568896 podStartE2EDuration="1.59568896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:02.593088727 +0000 UTC m=+944.224402204" watchObservedRunningTime="2026-11-03 21:04:02.59568896 +0000 UTC m=+944.227002438"
Nov 03 21:04:03.494221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.494188    5533 generic.go:356] "Generic (PLEG): container finished" podID="a168e771-2b8c-4c44-8cd7-f040996a98d1" containerID="b4a35b26abe0e8756ba90fcc276c400298b629a7763831393387423cb7042c76" exitCode=0
Nov 03 21:04:03.494718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.494245    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh" event={"ID":"a168e771-2b8c-4c44-8cd7-f040996a98d1","Type":"ContainerDied","Data":"b4a35b26abe0e8756ba90fcc276c400298b629a7763831393387423cb7042c76"}
Nov 03 21:04:03.780278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.780197    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"]
Nov 03 21:04:03.784395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.784373    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.823140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.823120    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknjb\" (UniqueName: \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.823247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.823151    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.823330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.823304    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.837559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.837527    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"]
Nov 03 21:04:03.924470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.924441    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.924613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.924516    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknjb\" (UniqueName: \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.924613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.924557    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.925130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.925111    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.926855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.926835    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:03.982264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:03.982238    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknjb\" (UniqueName: \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\") pod \"maas-api-revoke-sub-6c0b73fb-gvfjz\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:04.097537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.097471    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:04.467092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.467060    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"]
Nov 03 21:04:04.468070 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:04.468031    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12989a6_aad4_4c02_95be_887268f5d242.slice/crio-4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193 WatchSource:0}: Error finding container 4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193: Status 404 returned error can't find the container with id 4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193
Nov 03 21:04:04.502115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.502087    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz" event={"ID":"c12989a6-aad4-4c02-95be-887268f5d242","Type":"ContainerStarted","Data":"4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193"}
Nov 03 21:04:04.629879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.629854    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:04.734138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.734081    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\") pod \"a168e771-2b8c-4c44-8cd7-f040996a98d1\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") "
Nov 03 21:04:04.734256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.734183    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\") pod \"a168e771-2b8c-4c44-8cd7-f040996a98d1\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") "
Nov 03 21:04:04.734256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.734241    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\" (UniqueName: \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\") pod \"a168e771-2b8c-4c44-8cd7-f040996a98d1\" (UID: \"a168e771-2b8c-4c44-8cd7-f040996a98d1\") "
Nov 03 21:04:04.734492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.734469    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca" pod "a168e771-2b8c-4c44-8cd7-f040996a98d1" (UID: "a168e771-2b8c-4c44-8cd7-f040996a98d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.734584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.734571    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.736135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.736112    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5" pod "a168e771-2b8c-4c44-8cd7-f040996a98d1" (UID: "a168e771-2b8c-4c44-8cd7-f040996a98d1"). InnerVolumeSpecName "kube-api-access-flnx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.736234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.736142    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca" pod "a168e771-2b8c-4c44-8cd7-f040996a98d1" (UID: "a168e771-2b8c-4c44-8cd7-f040996a98d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:04.835560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.835535    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnx5\" (UniqueName: \"kubernetes.io/projected/a168e771-2b8c-4c44-8cd7-f040996a98d1-kube-api-access-flnx5\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.835560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:04.835557    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a168e771-2b8c-4c44-8cd7-f040996a98d1-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:05.509497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.509467    5533 generic.go:356] "Generic (PLEG): container finished" podID="c12989a6-aad4-4c02-95be-887268f5d242" containerID="4b949b2354806928fa7ac68d26b83d72bc87d758ffe164669ba41fdd99df8de8" exitCode=0
Nov 03 21:04:05.509849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.509551    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz" event={"ID":"c12989a6-aad4-4c02-95be-887268f5d242","Type":"ContainerDied","Data":"4b949b2354806928fa7ac68d26b83d72bc87d758ffe164669ba41fdd99df8de8"}
Nov 03 21:04:05.510902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.510879    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh" event={"ID":"a168e771-2b8c-4c44-8cd7-f040996a98d1","Type":"ContainerDied","Data":"6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494"}
Nov 03 21:04:05.510902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.510897    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"
Nov 03 21:04:05.511040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.510921    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b17c528d92654124c7962f2d312250a6ad5251c13344ca9264cfc701c348494"
Nov 03 21:04:05.706230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:05.706154    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:04:06.666829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.666806    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:06.762882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.762851    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\") pod \"c12989a6-aad4-4c02-95be-887268f5d242\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") "
Nov 03 21:04:06.763054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.762911    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\") pod \"c12989a6-aad4-4c02-95be-887268f5d242\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") "
Nov 03 21:04:06.763054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.762980    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\" (UniqueName: \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\") pod \"c12989a6-aad4-4c02-95be-887268f5d242\" (UID: \"c12989a6-aad4-4c02-95be-887268f5d242\") "
Nov 03 21:04:06.763291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.763250    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca" pod "c12989a6-aad4-4c02-95be-887268f5d242" (UID: "c12989a6-aad4-4c02-95be-887268f5d242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:06.765004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.764984    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca" pod "c12989a6-aad4-4c02-95be-887268f5d242" (UID: "c12989a6-aad4-4c02-95be-887268f5d242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:06.765089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.765047    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb" pod "c12989a6-aad4-4c02-95be-887268f5d242" (UID: "c12989a6-aad4-4c02-95be-887268f5d242"). InnerVolumeSpecName "kube-api-access-nknjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:06.864131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.864106    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nknjb\" (UniqueName: \"kubernetes.io/projected/c12989a6-aad4-4c02-95be-887268f5d242-kube-api-access-nknjb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:06.864131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.864129    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:06.864275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:06.864141    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12989a6-aad4-4c02-95be-887268f5d242-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:07.524160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:07.524130    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"
Nov 03 21:04:07.524347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:07.524138    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz" event={"ID":"c12989a6-aad4-4c02-95be-887268f5d242","Type":"ContainerDied","Data":"4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193"}
Nov 03 21:04:07.524347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:07.524276    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f4bbdd34a53038f36d8d1d7f695eed7bc7612b3fda205b7dbe73a63042d8193"
Nov 03 21:04:18.703970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.703932    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:04:18.704522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704501    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c12989a6-aad4-4c02-95be-887268f5d242" containerName="revoke-keys"
Nov 03 21:04:18.704608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704527    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12989a6-aad4-4c02-95be-887268f5d242" containerName="revoke-keys"
Nov 03 21:04:18.704608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704585    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a168e771-2b8c-4c44-8cd7-f040996a98d1" containerName="revoke-keys"
Nov 03 21:04:18.704608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704594    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168e771-2b8c-4c44-8cd7-f040996a98d1" containerName="revoke-keys"
Nov 03 21:04:18.704808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704712    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168e771-2b8c-4c44-8cd7-f040996a98d1" containerName="revoke-keys"
Nov 03 21:04:18.704808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.704727    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12989a6-aad4-4c02-95be-887268f5d242" containerName="revoke-keys"
Nov 03 21:04:18.709509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.709488    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.758928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.758905    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-791c32-openshift-default-dockercfg-zfm58\""
Nov 03 21:04:18.772601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772562    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvm6\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772610    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772672    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772724    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772754    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772781    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772819    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772862    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.772925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.772899    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.809914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.809872    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:04:18.874097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874061    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874105    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874163    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874199    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874233    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874256    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874372    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvm6\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874411    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874450    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.874926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.874900    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.875073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.875053    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.875149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.875091    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.875149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.875113    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.875415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.875398    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.876830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.876805    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.876922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.876908    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.932278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.932252    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:18.932491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:18.932472    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvm6\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:19.024688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.024662    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:19.264236 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:04:19.264138    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:04:19.441377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.441350    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:04:19.442726 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:19.442697    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9519b47f_00b3_4316_8620_0cf0b64af21d.slice/crio-2cb3b0093b212fb4cd7834ede71842e05bf6cf622017636126130af2a31a7438 WatchSource:0}: Error finding container 2cb3b0093b212fb4cd7834ede71842e05bf6cf622017636126130af2a31a7438: Status 404 returned error can't find the container with id 2cb3b0093b212fb4cd7834ede71842e05bf6cf622017636126130af2a31a7438
Nov 03 21:04:19.444967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.444932    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:19.445076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.445009    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:19.445076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.445055    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:19.597775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.597685    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" event={"ID":"9519b47f-00b3-4316-8620-0cf0b64af21d","Type":"ContainerStarted","Data":"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"}
Nov 03 21:04:19.597775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.597741    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" event={"ID":"9519b47f-00b3-4316-8620-0cf0b64af21d","Type":"ContainerStarted","Data":"2cb3b0093b212fb4cd7834ede71842e05bf6cf622017636126130af2a31a7438"}
Nov 03 21:04:19.714882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:19.714823    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podStartSLOduration=1.71480894 podStartE2EDuration="1.71480894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:19.712301797 +0000 UTC m=+961.343615275" watchObservedRunningTime="2026-11-03 21:04:19.71480894 +0000 UTC m=+961.346122416"
Nov 03 21:04:20.024872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:20.024840    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:20.026140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:20.026112    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:20.026282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:20.026171    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:21.025561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:21.025522    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:21.025992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:21.025583    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:22.025403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:22.025371    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:22.025559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:22.025428    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:23.025084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.025033    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:23.025464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.025136    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:23.182740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.182693    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:04:23.188176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.188155    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.219603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.219576    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.219740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.219644    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.219795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.219773    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.219795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.219790    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzcs\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.235285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.235251    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:04:23.282776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.279474    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-254f80d1d482\""
Nov 03 21:04:23.282776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.279759    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b3ede8887360\""
Nov 03 21:04:23.282776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.279888    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:04:23.282776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.279912    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-m9wt4\""
Nov 03 21:04:23.284774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.284713    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:04:23.284951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.284816    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:04:23.284951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.284884    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:04:23.321177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.321154    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.321297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.321219    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.321297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.321248    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzcs\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.321416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.321341    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.322110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.322085    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.322221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.322198    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.323843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.323818    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.390659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.389775    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzcs\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\") pod \"payload-processing-external-model-6d5f499cc7-mktgx\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") " pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.511916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.511879    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:23.926312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:23.926283    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:04:23.928135 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:23.928109    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa94ef76_306d_4e97_bc2e_d5a62e1b6644.slice/crio-f1f40b436da00868470d0b70ea8b09f39eee6158d01328c4f9c812bef050bae5 WatchSource:0}: Error finding container f1f40b436da00868470d0b70ea8b09f39eee6158d01328c4f9c812bef050bae5: Status 404 returned error can't find the container with id f1f40b436da00868470d0b70ea8b09f39eee6158d01328c4f9c812bef050bae5
Nov 03 21:04:24.024977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.024950    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:24.025079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.024998    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:24.634321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.634282    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" event={"ID":"fa94ef76-306d-4e97-bc2e-d5a62e1b6644","Type":"ContainerStarted","Data":"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"}
Nov 03 21:04:24.634820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.634337    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" event={"ID":"fa94ef76-306d-4e97-bc2e-d5a62e1b6644","Type":"ContainerStarted","Data":"f1f40b436da00868470d0b70ea8b09f39eee6158d01328c4f9c812bef050bae5"}
Nov 03 21:04:24.635063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.635043    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:24.746685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:24.745720    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" podStartSLOduration=1.745706819 podStartE2EDuration="1.745706819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:24.744822224 +0000 UTC m=+966.376135701" watchObservedRunningTime="2026-11-03 21:04:24.745706819 +0000 UTC m=+966.377020296"
Nov 03 21:04:25.025751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:25.025715    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 21:04:25.025900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:25.025762    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 21:04:26.129153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:26.129123    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:26.129623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:26.129457    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:26.130459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:26.130438    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:04:27.634425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.634392    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:04:27.637967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.637949    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.687528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.687500    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\""
Nov 03 21:04:27.738515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.738490    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:04:27.782560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782527    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.782820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782798    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.782929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782887    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shk6\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.782993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782928    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.782993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782962    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.783096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.782989    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.783166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.783150    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.783331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.783282    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.783716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.783557    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884444    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shk6\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884487    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884510    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884532    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884626    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884665    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884689    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884771    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.884928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884799    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.885245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.884938    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.885501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.885422    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.885501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.885437    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.885752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.885673    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.885930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.885911    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.887403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.887373    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.887544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.887526    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.942178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.942149    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.942796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.942766    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shk6\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:27.951340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:27.951318    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:28.373543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.373513    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:04:28.375237 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:28.375213    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995b0d00_5281_42a9_b8cd_9c544baab2cd.slice/crio-01fc674b5779e9299f2674668d9484a273fb654a52481894b9b2011024274882 WatchSource:0}: Error finding container 01fc674b5779e9299f2674668d9484a273fb654a52481894b9b2011024274882: Status 404 returned error can't find the container with id 01fc674b5779e9299f2674668d9484a273fb654a52481894b9b2011024274882
Nov 03 21:04:28.377941 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.377912    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:28.378019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.377975    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:28.378019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.378002    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:28.664155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.664058    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" event={"ID":"995b0d00-5281-42a9-b8cd-9c544baab2cd","Type":"ContainerStarted","Data":"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"}
Nov 03 21:04:28.664155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.664134    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" event={"ID":"995b0d00-5281-42a9-b8cd-9c544baab2cd","Type":"ContainerStarted","Data":"01fc674b5779e9299f2674668d9484a273fb654a52481894b9b2011024274882"}
Nov 03 21:04:28.779522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.779457    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podStartSLOduration=1.7794397069999999 podStartE2EDuration="1.779439707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:28.777520446 +0000 UTC m=+970.408833924" watchObservedRunningTime="2026-11-03 21:04:28.779439707 +0000 UTC m=+970.410753187"
Nov 03 21:04:28.959661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:28.958789    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:29.975811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:29.975774    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 21:04:29.976412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:29.976378    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 21:04:30.962967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:30.962930    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 21:04:30.963236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:30.963209    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 21:04:31.953994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:31.953955    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 21:04:31.954580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:31.954545    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 21:04:32.954661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:32.952978    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 21:04:32.954661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:32.953050    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 21:04:33.963660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:33.955987    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 21:04:33.963660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:33.956057    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 21:04:35.016482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:35.016451    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:35.017555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:35.017489    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:35.017804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:35.017776    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:04:36.650807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:36.650780    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:04:41.624617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.624588    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"]
Nov 03 21:04:41.628224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.628203    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.679568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.679537    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:41.720034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.720005    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.720160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.720072    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.720225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.720175    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksx4z\" (UniqueName: \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.726170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.726152    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:04:41.728792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.728772    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"]
Nov 03 21:04:41.821456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.821429    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.821597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.821483    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.821597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.821517    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksx4z\" (UniqueName: \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.822164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.822139    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.823852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.823827    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.878865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.878802    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksx4z\" (UniqueName: \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\") pod \"maas-api-revoke-sub-f6611650-5t6cv\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:41.940984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:41.940960    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:42.329112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:42.329075    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"]
Nov 03 21:04:42.330584 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:42.330561    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1eb4b1_0efd_4491_bfde_3016b95abd8c.slice/crio-a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948 WatchSource:0}: Error finding container a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948: Status 404 returned error can't find the container with id a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948
Nov 03 21:04:42.349842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:42.349815    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:04:42.766835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:42.766801    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv" event={"ID":"eb1eb4b1-0efd-4491-bfde-3016b95abd8c","Type":"ContainerStarted","Data":"151246e8a4be18e8e34b3373aa78255171f71aca71094fb3fa2b24704f21a218"}
Nov 03 21:04:42.767261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:42.766868    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv" event={"ID":"eb1eb4b1-0efd-4491-bfde-3016b95abd8c","Type":"ContainerStarted","Data":"a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948"}
Nov 03 21:04:42.876007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:42.875942    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv" podStartSLOduration=1.875926213 podStartE2EDuration="1.875926213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:42.87511676 +0000 UTC m=+984.506430244" watchObservedRunningTime="2026-11-03 21:04:42.875926213 +0000 UTC m=+984.507239691"
Nov 03 21:04:43.783796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:43.783763    5533 generic.go:356] "Generic (PLEG): container finished" podID="eb1eb4b1-0efd-4491-bfde-3016b95abd8c" containerID="151246e8a4be18e8e34b3373aa78255171f71aca71094fb3fa2b24704f21a218" exitCode=0
Nov 03 21:04:43.784525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:43.784497    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv" event={"ID":"eb1eb4b1-0efd-4491-bfde-3016b95abd8c","Type":"ContainerDied","Data":"151246e8a4be18e8e34b3373aa78255171f71aca71094fb3fa2b24704f21a218"}
Nov 03 21:04:44.083859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.083776    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:04:44.087456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.087434    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.200114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.200066    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:04:44.246367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246335    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246375    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246494    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246547    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246680    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246731    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246838    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246869    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.246961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.246942    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrb2\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347698    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347744    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347788    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrb2\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347844    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347881    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.347989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347952    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.347997    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348026    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348062    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348131    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348151    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348440    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.348936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.348911    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.350482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.350451    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.350586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.350571    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.405869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.405836    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.406624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.406585    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrb2\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-47j9m\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.700356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.700315    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:44.960536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:44.960505    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:45.058145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.058111    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\") pod \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") "
Nov 03 21:04:45.058293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.058223    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\" (UniqueName: \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\") pod \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") "
Nov 03 21:04:45.058387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.058368    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\") pod \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\" (UID: \"eb1eb4b1-0efd-4491-bfde-3016b95abd8c\") "
Nov 03 21:04:45.058534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.058506    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca" pod "eb1eb4b1-0efd-4491-bfde-3016b95abd8c" (UID: "eb1eb4b1-0efd-4491-bfde-3016b95abd8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:45.058791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.058762    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:45.060906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.060877    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z" pod "eb1eb4b1-0efd-4491-bfde-3016b95abd8c" (UID: "eb1eb4b1-0efd-4491-bfde-3016b95abd8c"). InnerVolumeSpecName "kube-api-access-ksx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:45.066953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.066924    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca" pod "eb1eb4b1-0efd-4491-bfde-3016b95abd8c" (UID: "eb1eb4b1-0efd-4491-bfde-3016b95abd8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:45.159572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.159535    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:45.159572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.159571    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksx4z\" (UniqueName: \"kubernetes.io/projected/eb1eb4b1-0efd-4491-bfde-3016b95abd8c-kube-api-access-ksx4z\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:45.190323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.190292    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:04:45.198501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.198469    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:45.199860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.199838    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:45.200012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.199996    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:45.817335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.817290    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" event={"ID":"db3ae28c-706c-4166-b11f-73bdaceeffe7","Type":"ContainerStarted","Data":"719b98a624140a8b4763eb05c10168a6c87cbad4ea8d38fb74ed2ee7efecca67"}
Nov 03 21:04:45.817542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.817356    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" event={"ID":"db3ae28c-706c-4166-b11f-73bdaceeffe7","Type":"ContainerStarted","Data":"6ecd47a1226967fdc2c6fccf677ac37956d0762ef8af9930f4b0821d8ee70622"}
Nov 03 21:04:45.818700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.818662    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"
Nov 03 21:04:45.818822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.818677    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv" event={"ID":"eb1eb4b1-0efd-4491-bfde-3016b95abd8c","Type":"ContainerDied","Data":"a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948"}
Nov 03 21:04:45.818822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.818795    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a331e63fbc4c9abbb9b4116bf17a66b6fa661faa9cd6a460bbb90463bb599948"
Nov 03 21:04:45.933081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:45.933025    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podStartSLOduration=1.933007653 podStartE2EDuration="1.933007653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:45.931725655 +0000 UTC m=+987.563039133" watchObservedRunningTime="2026-11-03 21:04:45.933007653 +0000 UTC m=+987.564321130"
Nov 03 21:04:46.701422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:46.701376    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:46.702966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:46.702936    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:46.703092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:46.703033    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:47.701672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:47.701625    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:47.702155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:47.701697    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:48.700730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:48.700690    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:48.700893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:48.700744    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:49.282160 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:04:49.282121    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:04:49.701021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:49.700991    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:49.701183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:49.701039    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:50.701570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:50.701533    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:50.701983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:50.701586    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:51.701614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:51.701574    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:51.702115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:51.701661    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:52.700651 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:52.700604    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-47j9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 21:04:52.700811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:52.700684    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 21:04:53.788928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:53.788891    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:53.789686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:53.789663    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:53.790364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:53.790344    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:04:59.186562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.186472    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 21:04:59.187136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.187116    5533 cpu_manager.go:414] "containerMap: removing container" podUID="eb1eb4b1-0efd-4491-bfde-3016b95abd8c" containerName="revoke-keys"
Nov 03 21:04:59.187217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.187138    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1eb4b1-0efd-4491-bfde-3016b95abd8c" containerName="revoke-keys"
Nov 03 21:04:59.187306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.187293    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1eb4b1-0efd-4491-bfde-3016b95abd8c" containerName="revoke-keys"
Nov 03 21:04:59.191939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.191915    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213575    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213610    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213650    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213753    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213778    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213806    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213918    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.213967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213953    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.214002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.213981    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x49\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.290458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.290418    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 21:04:59.340249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.340210    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.341417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.341394    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.341923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.341904    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.342472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.342451    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x49\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.343445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.341835    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.343918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.342371    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.344046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.341282    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.344156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.343066    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.344291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.344275    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.344774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.344754    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.345191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.345175    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.345728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.345709    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.346250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.344685    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.346425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.345106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.353286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.351695    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.354112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.354080    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.400589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.400558    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.400817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.400793    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x49\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-99g77\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.506420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.506322    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:04:59.931942 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:04:59.931915    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71ba5c1_7b5c_419e_88fa_65abf3e22cfc.slice/crio-67b001156105412eca7c88c80b8ba5950e02793f48ae62c436437ae044a7d8b7 WatchSource:0}: Error finding container 67b001156105412eca7c88c80b8ba5950e02793f48ae62c436437ae044a7d8b7: Status 404 returned error can't find the container with id 67b001156105412eca7c88c80b8ba5950e02793f48ae62c436437ae044a7d8b7
Nov 03 21:04:59.932269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.932243    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 21:04:59.935033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.935002    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:59.935119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.935063    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:04:59.935119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:04:59.935093    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:00.940688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:00.934999    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" event={"ID":"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc","Type":"ContainerStarted","Data":"8efc8eaef5a28dfdb6634b63ef36796df870b859f363d1f6a74c09db5321217d"}
Nov 03 21:05:00.940688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:00.935071    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" event={"ID":"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc","Type":"ContainerStarted","Data":"67b001156105412eca7c88c80b8ba5950e02793f48ae62c436437ae044a7d8b7"}
Nov 03 21:05:01.067118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.067049    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podStartSLOduration=2.067027982 podStartE2EDuration="2.067027982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:01.05521158 +0000 UTC m=+1002.686525058" watchObservedRunningTime="2026-11-03 21:05:01.067027982 +0000 UTC m=+1002.698341453"
Nov 03 21:05:01.508397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.508365    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:05:01.519834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.519804    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:01.521058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.520055    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:01.773715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.772821    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:05:01.794741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.793844    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.888655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.887932    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:05:01.941717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.941679    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2tc\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.942310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.942282    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.942459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.942441    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.942578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.942563    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.942740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.942723    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.942909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.942892    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.943027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.943012    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.943136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.943122    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:01.943335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:01.943317    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.044650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.044532    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2tc\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.044985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.044902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045013    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045036    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045551    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045629    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045800    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.045863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.045846    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046082    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046117    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046143    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046229    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046543    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.046990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.046786    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.048650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.048603    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.049854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.049829    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.115672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.112113    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.115672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.112733    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2tc\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.130398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.130369    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:02.507422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.507388    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:02.507596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.507440    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:02.604762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.604734    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:05:02.606327 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:05:02.606304    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7118dc_fcf0_4236_a976_3dac9fa643da.slice/crio-324c4f97286f7e4a54af54b0567550bf45e09780500fc68282888132f4228151 WatchSource:0}: Error finding container 324c4f97286f7e4a54af54b0567550bf45e09780500fc68282888132f4228151: Status 404 returned error can't find the container with id 324c4f97286f7e4a54af54b0567550bf45e09780500fc68282888132f4228151
Nov 03 21:05:02.608574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.608545    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:02.608677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.608605    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:02.608677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:02.608646    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:03.036287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.036250    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" event={"ID":"2f7118dc-fcf0-4236-a976-3dac9fa643da","Type":"ContainerStarted","Data":"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"}
Nov 03 21:05:03.036787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.036308    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" event={"ID":"2f7118dc-fcf0-4236-a976-3dac9fa643da","Type":"ContainerStarted","Data":"324c4f97286f7e4a54af54b0567550bf45e09780500fc68282888132f4228151"}
Nov 03 21:05:03.131099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.131061    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:03.132260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.132235    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:03.132394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.132282    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:03.156668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.152027    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podStartSLOduration=2.152008314 podStartE2EDuration="2.152008314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:03.149554063 +0000 UTC m=+1004.780867540" watchObservedRunningTime="2026-11-03 21:05:03.152008314 +0000 UTC m=+1004.783321793"
Nov 03 21:05:03.507432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.507392    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:03.507616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:03.507471    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:04.086627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.086589    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:05:04.098668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.097949    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.131722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.131693    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:04.131844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.131743    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:04.176768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.176733    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:05:04.212292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.212247    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.212526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.212506    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.212707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.212682    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.212872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.212848    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.213385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.213363    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcmf\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.213592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.213538    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.215288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.215248    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.215564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.215534    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.216009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.215991    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317482    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317550    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317568    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317607    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317621    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317665    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317688    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317712    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcmf\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.317851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317748    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.318006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.317982    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.318050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.318018    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.318269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.318248    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.318363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.318347    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.318470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.318453    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.320177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.320161    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.321116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.321100    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.400541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.399795    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.401766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.401732    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcmf\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.418228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.418208    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:04.507660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.507248    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:04.507660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.507314    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:04.907578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.907543    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:05:04.912271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.912232    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:04.912445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.912423    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:04.912527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:04.912490    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:05:05.058659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.057316    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" event={"ID":"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c","Type":"ContainerStarted","Data":"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"}
Nov 03 21:05:05.058659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.057370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" event={"ID":"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c","Type":"ContainerStarted","Data":"db1f143fe3192d25052f5f1fc0508ec55a4545ab2096209fc5dc679430090a59"}
Nov 03 21:05:05.132672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.132007    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:05.132672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.132065    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:05.179491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.178592    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podStartSLOduration=1.1785708320000001 podStartE2EDuration="1.178570832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:05.177948984 +0000 UTC m=+1006.809262462" watchObservedRunningTime="2026-11-03 21:05:05.178570832 +0000 UTC m=+1006.809884322"
Nov 03 21:05:05.418935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.418898    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:05.507109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.507023    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:05.507361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:05.507331    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:06.131981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.131706    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:06.131981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.131767    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:06.420207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.420165    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:06.420766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.420232    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:06.511700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.508784    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:06.512768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:06.512726    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:07.131751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.131721    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:07.131936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.131766    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:07.419091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.418996    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:07.419369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.419339    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:07.507042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.507005    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 21:05:07.507717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:07.507617    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 21:05:08.131684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:08.131622    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:08.131885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:08.131708    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:08.420720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:08.419999    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:08.420720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:08.420063    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:09.132188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.131793    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 21:05:09.132188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.131862    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 21:05:09.424504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.424437    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:09.424934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.424899    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:09.532811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.532776    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:05:09.532986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.532847    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:05:09.603098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.603059    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:05:09.604969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.604942    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:05:09.609229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:09.609211    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 21:05:10.204493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:10.204066    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:10.205674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:10.205627    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:10.206746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:10.206721    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:05:10.419821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:10.419781    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:10.419996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:10.419857    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:11.419128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:11.419092    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:05:11.419511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:11.419163    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:05:12.423408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:12.423378    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:12.423825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:12.423661    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:12.424398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:12.424383    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:05:19.263073 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:19.263033    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:05:20.528648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:20.528601    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:05:41.075543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:41.075511    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:05:49.264105 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:49.264063    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:05:51.250354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.250319    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:05:51.254453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.254433    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.352151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.352109    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-099a5e\"/\"e2e-conflict-b-3ba3a28c-kserve-self-signed-certs\""
Nov 03 21:05:51.354439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354410    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.354579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354457    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.354579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354476    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.354716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354584    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xqb\" (UniqueName: \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.354796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354778    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.354852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.354804    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.355434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.355413    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:05:51.456145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456109    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456166    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456185    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456216    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xqb\" (UniqueName: \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456295    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456313    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456568    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456618    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.456766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.456720    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.458521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.458499    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.458830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.458813    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.513490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.513439    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xqb\" (UniqueName: \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\") pod \"e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") " pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.558051 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:51.558025    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:51.558166 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:51.558057    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:51.558166 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:51.558147    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:05:52.058128555 +0000 UTC m=+1053.689442009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:51.568652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.568620    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:51.945524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:51.945499    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:05:51.946198 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:05:51.946169    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ce81338_1f47_4aad_a714_986a1217bb6d.slice/crio-543daf212b8597c139c42edd7b80fbd21d0a6ad6287e935cf9f084e0e2bce7fc WatchSource:0}: Error finding container 543daf212b8597c139c42edd7b80fbd21d0a6ad6287e935cf9f084e0e2bce7fc: Status 404 returned error can't find the container with id 543daf212b8597c139c42edd7b80fbd21d0a6ad6287e935cf9f084e0e2bce7fc
Nov 03 21:05:52.068985 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:52.068947    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:52.068985 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:52.068979    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:52.069155 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:52.069045    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:05:53.069025143 +0000 UTC m=+1054.700338614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:52.415257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:52.415152    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" event={"ID":"5ce81338-1f47-4aad-a714-986a1217bb6d","Type":"ContainerStarted","Data":"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"}
Nov 03 21:05:52.415257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:52.415207    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" event={"ID":"5ce81338-1f47-4aad-a714-986a1217bb6d","Type":"ContainerStarted","Data":"543daf212b8597c139c42edd7b80fbd21d0a6ad6287e935cf9f084e0e2bce7fc"}
Nov 03 21:05:52.527354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:52.527292    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" podStartSLOduration=1.52727892 podStartE2EDuration="1.52727892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:52.525133431 +0000 UTC m=+1054.156446907" watchObservedRunningTime="2026-11-03 21:05:52.52727892 +0000 UTC m=+1054.158592438"
Nov 03 21:05:52.569817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:05:52.569782    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:05:53.079728 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:53.079691    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:53.079728 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:53.079722    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:53.079971 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:53.079785    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:05:55.079766103 +0000 UTC m=+1056.711079572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:55.099037 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:55.099004    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:55.099037 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:55.099038    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:55.099462 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:55.099102    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:05:59.099083409 +0000 UTC m=+1060.730396884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:59.141121 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:59.141092    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:59.141121 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:59.141120    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:05:59.141546 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:05:59.141174    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:06:07.14116125 +0000 UTC m=+1068.772474705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:01.569674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:01.569628    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:06:01.582277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:01.582255    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:06:02.308887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:02.308851    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:06:02.498100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:02.498066    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:06:07.240554 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:07.240520    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:07.240554 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:07.240551    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:07.241143 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:07.240614    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:06:23.240597477 +0000 UTC m=+1084.871910942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:09.564502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.564463    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:06:09.568240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.568215    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.617138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.617110    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0652b81d8bc9\""
Nov 03 21:06:09.617281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.617192    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6a6e49620b7d\""
Nov 03 21:06:09.664250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.664218    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.664428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.664329    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.664497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.664478    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.664534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.664518    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgmt\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.666862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.666842    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:06:09.765581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.765549    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.765715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.765599    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgmt\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.765715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.765704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.765851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.765767    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.766193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.766164    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.766465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.766446    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.768240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.768224    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.823680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.823614    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgmt\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\") pod \"payload-processing-external-model-59d447b455-z7x49\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") " pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:09.881651 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:09.881607    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:10.259780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:10.258179    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:06:10.625438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:10.625358    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" event={"ID":"1998af65-866e-4c04-b0a8-ccd0ca6f494d","Type":"ContainerStarted","Data":"c721fcf04fcf6deba2b600bde91640eaa6ab2cc4a4146009bbd2bde1bf40964f"}
Nov 03 21:06:10.625799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:10.625518    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:10.625799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:10.625548    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" event={"ID":"1998af65-866e-4c04-b0a8-ccd0ca6f494d","Type":"ContainerStarted","Data":"785314e4a28785a74bead9987caf18bff03f67e048d5895d4018da4ed667b2e0"}
Nov 03 21:06:10.735833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:10.735773    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" podStartSLOduration=1.7357564060000001 podStartE2EDuration="1.735756406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:10.73477389 +0000 UTC m=+1072.366087370" watchObservedRunningTime="2026-11-03 21:06:10.735756406 +0000 UTC m=+1072.367069886"
Nov 03 21:06:18.669301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:18.669270    5533 scope.go:122] "RemoveContainer" containerID="27ae5f2a74889f35ba74e90001e1c6225dfa7ab117bd723a4af0d706cfbf2d35"
Nov 03 21:06:18.678981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:18.678960    5533 scope.go:122] "RemoveContainer" containerID="aa743326ade3a72e6b3157964f2e3f510ee39402f77e6340e2a86442dfb4213d"
Nov 03 21:06:18.688699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:18.688651    5533 scope.go:122] "RemoveContainer" containerID="ac416411dae7a72f4c89b5c2d16025cdd6b88a50508ccd4b7c1bac208a11ead2"
Nov 03 21:06:18.696521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:18.696502    5533 scope.go:122] "RemoveContainer" containerID="68b3827f3fbafb77a3aef70b8c0a00e52ca144ca17382c36512604d5e5478d61"
Nov 03 21:06:19.260582 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:19.260547    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:06:21.634327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:21.634297    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:06:21.793892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:21.793863    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:06:23.318973 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:23.318940    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-254f80d1d482: secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:23.318973 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:23.318969    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx: secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:23.319416 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:23.319035    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials podName:fa94ef76-306d-4e97-bc2e-d5a62e1b6644 nodeName:}" failed. No retries permitted until 2026-11-03 21:06:55.319017533 +0000 UTC m=+1116.950331007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials") pod "payload-processing-external-model-6d5f499cc7-mktgx" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644") : secret "agc-cred-254f80d1d482" not found
Nov 03 21:06:26.794982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:26.794933    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" podUID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" containerName="payload-processing" containerID="cri-o://b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326" gracePeriod=25
Nov 03 21:06:27.048759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.048706    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:06:27.163716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.163684    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\") pod \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") "
Nov 03 21:06:27.164288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.163740    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\") pod \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") "
Nov 03 21:06:27.164288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.163793    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\") pod \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") "
Nov 03 21:06:27.164288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.163963    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\") pod \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\" (UID: \"fa94ef76-306d-4e97-bc2e-d5a62e1b6644\") "
Nov 03 21:06:27.164288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.164272    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume" pod "fa94ef76-306d-4e97-bc2e-d5a62e1b6644" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:27.164595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.164352    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:27.164595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.164405    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume" pod "fa94ef76-306d-4e97-bc2e-d5a62e1b6644" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:27.167565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.167540    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs" pod "fa94ef76-306d-4e97-bc2e-d5a62e1b6644" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644"). InnerVolumeSpecName "kube-api-access-4lzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:27.167862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.167836    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials" pod "fa94ef76-306d-4e97-bc2e-d5a62e1b6644" (UID: "fa94ef76-306d-4e97-bc2e-d5a62e1b6644"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:27.265325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.265293    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:27.265325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.265326    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:27.265482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.265341    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzcs\" (UniqueName: \"kubernetes.io/projected/fa94ef76-306d-4e97-bc2e-d5a62e1b6644-kube-api-access-4lzcs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:27.784662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.784604    5533 generic.go:356] "Generic (PLEG): container finished" podID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" containerID="b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326" exitCode=0
Nov 03 21:06:27.784853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.784691    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"
Nov 03 21:06:27.784853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.784691    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" event={"ID":"fa94ef76-306d-4e97-bc2e-d5a62e1b6644","Type":"ContainerDied","Data":"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"}
Nov 03 21:06:27.784853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.784748    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx" event={"ID":"fa94ef76-306d-4e97-bc2e-d5a62e1b6644","Type":"ContainerDied","Data":"f1f40b436da00868470d0b70ea8b09f39eee6158d01328c4f9c812bef050bae5"}
Nov 03 21:06:27.784853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.784776    5533 scope.go:122] "RemoveContainer" containerID="b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"
Nov 03 21:06:27.796234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.796007    5533 scope.go:122] "RemoveContainer" containerID="b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"
Nov 03 21:06:27.796527 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:27.796276    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326\": container with ID starting with b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326 not found: ID does not exist" containerID="b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"
Nov 03 21:06:27.796527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.796299    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326"} err="failed to get container status \"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326\": rpc error: code = NotFound desc = could not find container \"b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326\": container with ID starting with b1fe3f7973d80e9eefc20debf3daddd8daad4c0947f264c6a6b2b4807b7a2326 not found: ID does not exist"
Nov 03 21:06:27.944386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.944353    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:06:27.987685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:27.987631    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d5f499cc7-mktgx"]
Nov 03 21:06:28.494779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:28.494745    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" path="/var/lib/kubelet/pods/fa94ef76-306d-4e97-bc2e-d5a62e1b6644/volumes"
Nov 03 21:06:28.549748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:28.549714    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:06:49.261217 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:06:49.261181    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:06:55.950199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.950165    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 21:06:55.950850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.950829    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" containerName="payload-processing"
Nov 03 21:06:55.950954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.950850    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" containerName="payload-processing"
Nov 03 21:06:55.951030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.951014    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa94ef76-306d-4e97-bc2e-d5a62e1b6644" containerName="payload-processing"
Nov 03 21:06:55.954400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.954368    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992234    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992271    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992310    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992475    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992502    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:55.992525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:55.992519    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x9r\" (UniqueName: \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.050429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.050398    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"distinct-w3-320f60-kserve-self-signed-certs\""
Nov 03 21:06:56.053606 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.053581    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 21:06:56.093226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093196    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.093226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093245    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.093226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093273    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x9r\" (UniqueName: \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093324    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093354    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093420    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093826    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.093882    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.094477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.094235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.096870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.096704    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.096870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.096833    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.151969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.151935    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x9r\" (UniqueName: \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\") pod \"distinct-w3-320f60-kserve-58bd67b8c-bnpg8\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.265717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.265622    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:06:56.633197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:56.633168    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 21:06:56.634764 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:06:56.634741    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod651af5cf_fbb4_4ade_b468_d1de7b667163.slice/crio-f9aedbe2d7ce81b64875e36b7db86ae3758c840b54e28ea48a91cbb44073818b WatchSource:0}: Error finding container f9aedbe2d7ce81b64875e36b7db86ae3758c840b54e28ea48a91cbb44073818b: Status 404 returned error can't find the container with id f9aedbe2d7ce81b64875e36b7db86ae3758c840b54e28ea48a91cbb44073818b
Nov 03 21:06:57.009079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:57.009046    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" event={"ID":"651af5cf-fbb4-4ade-b468-d1de7b667163","Type":"ContainerStarted","Data":"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"}
Nov 03 21:06:57.009547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:57.009093    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" event={"ID":"651af5cf-fbb4-4ade-b468-d1de7b667163","Type":"ContainerStarted","Data":"f9aedbe2d7ce81b64875e36b7db86ae3758c840b54e28ea48a91cbb44073818b"}
Nov 03 21:06:57.121117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:57.121060    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podStartSLOduration=2.121042544 podStartE2EDuration="2.121042544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:57.119365555 +0000 UTC m=+1118.750679023" watchObservedRunningTime="2026-11-03 21:06:57.121042544 +0000 UTC m=+1118.752356024"
Nov 03 21:06:57.266751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:06:57.266628    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:07:03.505739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:03.505703    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:07:04.055524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:04.055493    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:07:06.266537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:06.266499    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:07:06.279811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:06.279784    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:07:07.080732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:07.080696    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:07:19.075692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.075589    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" podUID="5ce81338-1f47-4aad-a714-986a1217bb6d" containerName="main" containerID="cri-o://2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7" gracePeriod=45
Nov 03 21:07:19.273575 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:07:19.273482    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:07:19.352312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.352289    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:07:19.358729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358705    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\" (UniqueName: \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.358793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358767    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.358843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358827    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.358887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358861    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.358931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358890    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.358989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.358953    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\") pod \"5ce81338-1f47-4aad-a714-986a1217bb6d\" (UID: \"5ce81338-1f47-4aad-a714-986a1217bb6d\") "
Nov 03 21:07:19.359082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359042    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:19.359205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359174    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:19.359294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359268    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:19.359573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359550    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:19.359701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359603    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:19.359701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.359620    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:19.361074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.361040    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:07:19.361074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.361065    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "kube-api-access-x2xqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:19.361200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.361114    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs" pod "5ce81338-1f47-4aad-a714-986a1217bb6d" (UID: "5ce81338-1f47-4aad-a714-986a1217bb6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:19.460103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.460077    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xqb\" (UniqueName: \"kubernetes.io/projected/5ce81338-1f47-4aad-a714-986a1217bb6d-kube-api-access-x2xqb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:19.460103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.460102    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ce81338-1f47-4aad-a714-986a1217bb6d-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:19.460225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:19.460114    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ce81338-1f47-4aad-a714-986a1217bb6d-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:20.190892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.190857    5533 generic.go:356] "Generic (PLEG): container finished" podID="5ce81338-1f47-4aad-a714-986a1217bb6d" containerID="2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7" exitCode=0
Nov 03 21:07:20.191548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.191341    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" event={"ID":"5ce81338-1f47-4aad-a714-986a1217bb6d","Type":"ContainerDied","Data":"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"}
Nov 03 21:07:20.191687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.191590    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b" event={"ID":"5ce81338-1f47-4aad-a714-986a1217bb6d","Type":"ContainerDied","Data":"543daf212b8597c139c42edd7b80fbd21d0a6ad6287e935cf9f084e0e2bce7fc"}
Nov 03 21:07:20.191687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.191625    5533 scope.go:122] "RemoveContainer" containerID="2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"
Nov 03 21:07:20.191810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.191379    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"
Nov 03 21:07:20.213929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.213899    5533 scope.go:122] "RemoveContainer" containerID="2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"
Nov 03 21:07:20.214251 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:07:20.214228    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7\": container with ID starting with 2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7 not found: ID does not exist" containerID="2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"
Nov 03 21:07:20.214343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.214262    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7"} err="failed to get container status \"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7\": rpc error: code = NotFound desc = could not find container \"2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7\": container with ID starting with 2749a51c422fa48d81982ffd68bad35e66bc6df2efa70a4970d58b57397ef5f7 not found: ID does not exist"
Nov 03 21:07:20.350560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.350532    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:07:20.396897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.396867    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/e2e-conflict-b-3ba3a28c-kserve-867c875856-b665b"]
Nov 03 21:07:20.492961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.492893    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce81338-1f47-4aad-a714-986a1217bb6d" path="/var/lib/kubelet/pods/5ce81338-1f47-4aad-a714-986a1217bb6d/volumes"
Nov 03 21:07:20.909941 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.909865    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"]
Nov 03 21:07:20.910339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.910325    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5ce81338-1f47-4aad-a714-986a1217bb6d" containerName="main"
Nov 03 21:07:20.910381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.910339    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce81338-1f47-4aad-a714-986a1217bb6d" containerName="main"
Nov 03 21:07:20.910431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.910422    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce81338-1f47-4aad-a714-986a1217bb6d" containerName="main"
Nov 03 21:07:20.914932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.914914    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:20.985461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.985436    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:20.985582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.985480    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckj8\" (UniqueName: \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:20.985582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:20.985501    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.011557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.011528    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:07:21.011690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.011667    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:07:21.014643 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.014614    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"]
Nov 03 21:07:21.086595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.086571    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.086721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.086611    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckj8\" (UniqueName: \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.086721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.086630    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.087249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.087223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.088844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.088824    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.143902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.143880    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckj8\" (UniqueName: \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\") pod \"maas-api-revoke-sub-f67b311d-zbrbt\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.228664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.228631    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:21.599007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:21.598971    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"]
Nov 03 21:07:21.600487 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:21.600437    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d36462a_0ffa_45f1_9052_5ffe36404afd.slice/crio-f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3 WatchSource:0}: Error finding container f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3: Status 404 returned error can't find the container with id f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3
Nov 03 21:07:22.209344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:22.209306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt" event={"ID":"8d36462a-0ffa-45f1-9052-5ffe36404afd","Type":"ContainerStarted","Data":"57143f482a8d553078c6c32a0d367fc3ab35ef81d32a6afdcc569d511922d0cf"}
Nov 03 21:07:22.209506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:22.209369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt" event={"ID":"8d36462a-0ffa-45f1-9052-5ffe36404afd","Type":"ContainerStarted","Data":"f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3"}
Nov 03 21:07:22.318555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:22.318509    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt" podStartSLOduration=2.31849405 podStartE2EDuration="2.31849405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:22.318058811 +0000 UTC m=+1143.949372287" watchObservedRunningTime="2026-11-03 21:07:22.31849405 +0000 UTC m=+1143.949807528"
Nov 03 21:07:23.217348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:23.217308    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d36462a-0ffa-45f1-9052-5ffe36404afd" containerID="57143f482a8d553078c6c32a0d367fc3ab35ef81d32a6afdcc569d511922d0cf" exitCode=0
Nov 03 21:07:23.217526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:23.217384    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt" event={"ID":"8d36462a-0ffa-45f1-9052-5ffe36404afd","Type":"ContainerDied","Data":"57143f482a8d553078c6c32a0d367fc3ab35ef81d32a6afdcc569d511922d0cf"}
Nov 03 21:07:24.357238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.357212    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:24.428655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.428610    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\") pod \"8d36462a-0ffa-45f1-9052-5ffe36404afd\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") "
Nov 03 21:07:24.428802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.428687    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\") pod \"8d36462a-0ffa-45f1-9052-5ffe36404afd\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") "
Nov 03 21:07:24.428802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.428737    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\" (UniqueName: \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\") pod \"8d36462a-0ffa-45f1-9052-5ffe36404afd\" (UID: \"8d36462a-0ffa-45f1-9052-5ffe36404afd\") "
Nov 03 21:07:24.429074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.429045    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca" pod "8d36462a-0ffa-45f1-9052-5ffe36404afd" (UID: "8d36462a-0ffa-45f1-9052-5ffe36404afd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:24.429221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.429201    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:24.430707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.430678    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca" pod "8d36462a-0ffa-45f1-9052-5ffe36404afd" (UID: "8d36462a-0ffa-45f1-9052-5ffe36404afd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:24.430878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.430848    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8" pod "8d36462a-0ffa-45f1-9052-5ffe36404afd" (UID: "8d36462a-0ffa-45f1-9052-5ffe36404afd"). InnerVolumeSpecName "kube-api-access-rckj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:24.530754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.530727    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d36462a-0ffa-45f1-9052-5ffe36404afd-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:24.530754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:24.530754    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rckj8\" (UniqueName: \"kubernetes.io/projected/8d36462a-0ffa-45f1-9052-5ffe36404afd-kube-api-access-rckj8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:25.230578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:25.230549    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"
Nov 03 21:07:25.230753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:25.230549    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt" event={"ID":"8d36462a-0ffa-45f1-9052-5ffe36404afd","Type":"ContainerDied","Data":"f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3"}
Nov 03 21:07:25.230753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:25.230683    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f027dffdcb83e536312661a5991e024fdf008b78b16a68ad76c5b695c3c999c3"
Nov 03 21:07:28.736451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:28.736418    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:07:33.219449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.219415    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:07:33.219930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.219915    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d36462a-0ffa-45f1-9052-5ffe36404afd" containerName="revoke-keys"
Nov 03 21:07:33.220011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.219931    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d36462a-0ffa-45f1-9052-5ffe36404afd" containerName="revoke-keys"
Nov 03 21:07:33.220059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.220018    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d36462a-0ffa-45f1-9052-5ffe36404afd" containerName="revoke-keys"
Nov 03 21:07:33.223580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.223564    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.272886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.272863    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-320f60-openshift-default-dockercfg-jct6g\""
Nov 03 21:07:33.322490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.322466    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:07:33.328902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.328879    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.328913    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.328932    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.328955    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.329018    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.329126    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.329176    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.329253    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjhv\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.329382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.329362    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430058    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430098    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430115    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjhv\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430157    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430211    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430337    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430377    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430418    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430445    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430524    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430588    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430706    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.430832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.430781    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.431122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.431097    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.432480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.432459    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.432748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.432721    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.487692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.487615    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.487823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.487805    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjhv\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-9mx87\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.536801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.536776    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:33.958392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.958357    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:07:33.958806 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:33.958775    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad77ce8_9e3f_4630_916e_d6dafd597c2a.slice/crio-432cea5eeed322558caea09facf6ce09ed83f9c482239034710c246a368b7570 WatchSource:0}: Error finding container 432cea5eeed322558caea09facf6ce09ed83f9c482239034710c246a368b7570: Status 404 returned error can't find the container with id 432cea5eeed322558caea09facf6ce09ed83f9c482239034710c246a368b7570
Nov 03 21:07:33.961237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.961206    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:33.961329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.961272    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:33.961329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:33.961299    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:34.293370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:34.293286    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" event={"ID":"8ad77ce8-9e3f-4630-916e-d6dafd597c2a","Type":"ContainerStarted","Data":"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"}
Nov 03 21:07:34.293370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:34.293342    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" event={"ID":"8ad77ce8-9e3f-4630-916e-d6dafd597c2a","Type":"ContainerStarted","Data":"432cea5eeed322558caea09facf6ce09ed83f9c482239034710c246a368b7570"}
Nov 03 21:07:34.407989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:34.407936    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podStartSLOduration=1.407923506 podStartE2EDuration="1.407923506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:34.406459187 +0000 UTC m=+1156.037772666" watchObservedRunningTime="2026-11-03 21:07:34.407923506 +0000 UTC m=+1156.039237047"
Nov 03 21:07:34.536984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:34.536951    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:35.538712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:35.538666    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:35.539123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:35.538728    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:36.415276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.415225    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"]
Nov 03 21:07:36.419188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.419163    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.472015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.471986    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:07:36.477898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.477808    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"]
Nov 03 21:07:36.518823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.518793    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:07:36.537141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.537112    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:36.537268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.537163    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:36.584280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.584242    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.584677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.584391    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptk7\" (UniqueName: \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.584677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.584500    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.685590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.685559    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.685769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.685626    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptk7\" (UniqueName: \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.685769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.685742    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.686219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.686197    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.688028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.688002    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:36.743565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:36.743542    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptk7\" (UniqueName: \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\") pod \"maas-api-revoke-sub-01908125-7228k\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:37.033578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:37.033493    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:37.438330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:37.438295    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"]
Nov 03 21:07:37.438582 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:37.438555    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd72f58_5d6a_4201_95d0_d49a46e81d75.slice/crio-c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0 WatchSource:0}: Error finding container c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0: Status 404 returned error can't find the container with id c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0
Nov 03 21:07:37.537873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:37.537829    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:37.537977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:37.537892    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:38.166149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.166060    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"]
Nov 03 21:07:38.181655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.177850    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.217119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.217092    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"]
Nov 03 21:07:38.303218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.303186    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.303411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.303392    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5wh\" (UniqueName: \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.303460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.303435    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.319427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.319397    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k" event={"ID":"fdd72f58-5d6a-4201-95d0-d49a46e81d75","Type":"ContainerStarted","Data":"501449f4dab3a32e6695adfa358abb8cb381ff6a701a7088b0a4fa400c9f5c14"}
Nov 03 21:07:38.319569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.319456    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k" event={"ID":"fdd72f58-5d6a-4201-95d0-d49a46e81d75","Type":"ContainerStarted","Data":"c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0"}
Nov 03 21:07:38.404730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.404700    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.404911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.404828    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5wh\" (UniqueName: \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.404911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.404853    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.405419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.405394    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.407165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.407142    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.442795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.442753    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k" podStartSLOduration=2.44274116 podStartE2EDuration="2.44274116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:38.441723445 +0000 UTC m=+1160.073036939" watchObservedRunningTime="2026-11-03 21:07:38.44274116 +0000 UTC m=+1160.074054636"
Nov 03 21:07:38.482279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.482253    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5wh\" (UniqueName: \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\") pod \"maas-api-revoke-sub-8edece9f-4h788\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.498232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.498211    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:38.539106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.537453    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:38.539106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.537513    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:38.905135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:38.905105    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"]
Nov 03 21:07:38.905310 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:38.905287    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4237e068_0c54_4243_a84c_59570b99134d.slice/crio-95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2 WatchSource:0}: Error finding container 95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2: Status 404 returned error can't find the container with id 95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2
Nov 03 21:07:39.334919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.327139    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788" event={"ID":"4237e068-0c54-4243-a84c-59570b99134d","Type":"ContainerStarted","Data":"7dc590ffb2398082f8b4d26080644491916c689f36f29305b07a4bed5e95dcd8"}
Nov 03 21:07:39.334919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.327675    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788" event={"ID":"4237e068-0c54-4243-a84c-59570b99134d","Type":"ContainerStarted","Data":"95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2"}
Nov 03 21:07:39.334919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.329908    5533 generic.go:356] "Generic (PLEG): container finished" podID="fdd72f58-5d6a-4201-95d0-d49a46e81d75" containerID="501449f4dab3a32e6695adfa358abb8cb381ff6a701a7088b0a4fa400c9f5c14" exitCode=0
Nov 03 21:07:39.334919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.330005    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k" event={"ID":"fdd72f58-5d6a-4201-95d0-d49a46e81d75","Type":"ContainerDied","Data":"501449f4dab3a32e6695adfa358abb8cb381ff6a701a7088b0a4fa400c9f5c14"}
Nov 03 21:07:39.440214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.440161    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788" podStartSLOduration=1.4401481870000001 podStartE2EDuration="1.440148187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:39.437134606 +0000 UTC m=+1161.068448084" watchObservedRunningTime="2026-11-03 21:07:39.440148187 +0000 UTC m=+1161.071461663"
Nov 03 21:07:39.538408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.538374    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:39.538564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:39.538423    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:40.338228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.338194    5533 generic.go:356] "Generic (PLEG): container finished" podID="4237e068-0c54-4243-a84c-59570b99134d" containerID="7dc590ffb2398082f8b4d26080644491916c689f36f29305b07a4bed5e95dcd8" exitCode=0
Nov 03 21:07:40.338705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.338235    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788" event={"ID":"4237e068-0c54-4243-a84c-59570b99134d","Type":"ContainerDied","Data":"7dc590ffb2398082f8b4d26080644491916c689f36f29305b07a4bed5e95dcd8"}
Nov 03 21:07:40.505415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.505392    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:40.537845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.537821    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 21:07:40.537965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.537869    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 21:07:40.629213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.629121    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\") pod \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") "
Nov 03 21:07:40.629213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.629197    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\" (UniqueName: \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\") pod \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") "
Nov 03 21:07:40.629597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.629561    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca" pod "fdd72f58-5d6a-4201-95d0-d49a46e81d75" (UID: "fdd72f58-5d6a-4201-95d0-d49a46e81d75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:40.629597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.629568    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\") pod \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\" (UID: \"fdd72f58-5d6a-4201-95d0-d49a46e81d75\") "
Nov 03 21:07:40.630158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.630131    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.632154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.632125    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca" pod "fdd72f58-5d6a-4201-95d0-d49a46e81d75" (UID: "fdd72f58-5d6a-4201-95d0-d49a46e81d75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:40.635295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.635265    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7" pod "fdd72f58-5d6a-4201-95d0-d49a46e81d75" (UID: "fdd72f58-5d6a-4201-95d0-d49a46e81d75"). InnerVolumeSpecName "kube-api-access-8ptk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:40.731823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.731794    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptk7\" (UniqueName: \"kubernetes.io/projected/fdd72f58-5d6a-4201-95d0-d49a46e81d75-kube-api-access-8ptk7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:40.731823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:40.731822    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd72f58-5d6a-4201-95d0-d49a46e81d75-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:41.345446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.345414    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"
Nov 03 21:07:41.345958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.345456    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k" event={"ID":"fdd72f58-5d6a-4201-95d0-d49a46e81d75","Type":"ContainerDied","Data":"c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0"}
Nov 03 21:07:41.345958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.345504    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6031d46d10464664a3c7734632d65a7dbd2b4f597b3294115fc1208e20e7ba0"
Nov 03 21:07:41.515629 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.515602    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:41.542870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.542842    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:41.543256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.543223    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:41.543946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.543924    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:07:41.643066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.642989    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\") pod \"4237e068-0c54-4243-a84c-59570b99134d\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") "
Nov 03 21:07:41.643222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.643106    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\" (UniqueName: \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\") pod \"4237e068-0c54-4243-a84c-59570b99134d\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") "
Nov 03 21:07:41.643222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.643176    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\") pod \"4237e068-0c54-4243-a84c-59570b99134d\" (UID: \"4237e068-0c54-4243-a84c-59570b99134d\") "
Nov 03 21:07:41.643564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.643535    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca" pod "4237e068-0c54-4243-a84c-59570b99134d" (UID: "4237e068-0c54-4243-a84c-59570b99134d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:41.643733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.643706    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4237e068-0c54-4243-a84c-59570b99134d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:41.645243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.645224    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca" pod "4237e068-0c54-4243-a84c-59570b99134d" (UID: "4237e068-0c54-4243-a84c-59570b99134d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:41.645474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.645449    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh" pod "4237e068-0c54-4243-a84c-59570b99134d" (UID: "4237e068-0c54-4243-a84c-59570b99134d"). InnerVolumeSpecName "kube-api-access-vd5wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:41.744258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.744233    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4237e068-0c54-4243-a84c-59570b99134d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:41.744369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:41.744260    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5wh\" (UniqueName: \"kubernetes.io/projected/4237e068-0c54-4243-a84c-59570b99134d-kube-api-access-vd5wh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:42.353473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:42.353431    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"
Nov 03 21:07:42.353473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:42.353442    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788" event={"ID":"4237e068-0c54-4243-a84c-59570b99134d","Type":"ContainerDied","Data":"95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2"}
Nov 03 21:07:42.354027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:42.353513    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ff625fa782d2f88c1b0ef3416f1bedf72102b0c60d9ba72c2c0a34218a2be2"
Nov 03 21:07:48.880689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.880653    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"]
Nov 03 21:07:48.881142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881120    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fdd72f58-5d6a-4201-95d0-d49a46e81d75" containerName="revoke-keys"
Nov 03 21:07:48.881142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881135    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd72f58-5d6a-4201-95d0-d49a46e81d75" containerName="revoke-keys"
Nov 03 21:07:48.881142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881145    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4237e068-0c54-4243-a84c-59570b99134d" containerName="revoke-keys"
Nov 03 21:07:48.881359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881151    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4237e068-0c54-4243-a84c-59570b99134d" containerName="revoke-keys"
Nov 03 21:07:48.881359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881248    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4237e068-0c54-4243-a84c-59570b99134d" containerName="revoke-keys"
Nov 03 21:07:48.881359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.881264    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd72f58-5d6a-4201-95d0-d49a46e81d75" containerName="revoke-keys"
Nov 03 21:07:48.884624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.884608    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:48.920554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.920528    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:48.920695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.920667    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpmj\" (UniqueName: \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:48.920757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.920742    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:48.936998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.936932    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:07:48.983473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.983446    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:07:48.987688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:48.987668    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"]
Nov 03 21:07:49.021347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.021318    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpmj\" (UniqueName: \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.021456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.021377    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.021456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.021448    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.022025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.022003    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.023698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.023681    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.078899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.078870    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpmj\" (UniqueName: \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\") pod \"maas-api-revoke-sub-392c0f24-mchtd\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.200178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.200145    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:49.279777 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:07:49.279324    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:07:49.574649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:49.574607    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"]
Nov 03 21:07:49.575838 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:49.575797    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf542f4b0_039c_40e0_802c_e67f5f474704.slice/crio-d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e WatchSource:0}: Error finding container d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e: Status 404 returned error can't find the container with id d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e
Nov 03 21:07:50.415525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:50.415496    5533 generic.go:356] "Generic (PLEG): container finished" podID="f542f4b0-039c-40e0-802c-e67f5f474704" containerID="c1dbf034c8180e1448bbee5733ffebb7949800ebc93758d7c29e3d31837cb553" exitCode=0
Nov 03 21:07:50.415884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:50.415587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd" event={"ID":"f542f4b0-039c-40e0-802c-e67f5f474704","Type":"ContainerDied","Data":"c1dbf034c8180e1448bbee5733ffebb7949800ebc93758d7c29e3d31837cb553"}
Nov 03 21:07:50.415884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:50.415657    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd" event={"ID":"f542f4b0-039c-40e0-802c-e67f5f474704","Type":"ContainerStarted","Data":"d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e"}
Nov 03 21:07:51.228541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.228512    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:07:51.232227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.232206    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.332157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.332129    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:07:51.343907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.343874    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.343932    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344055    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344108    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344252    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344289    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwvr\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344425    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344471    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.344560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.344541    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445141    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445187    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwvr\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445262    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445290    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445320    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445352    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445387    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445751    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.445836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.445798    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.448058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.446262    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.448058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.446296    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.448058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.446550    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.448058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.446814    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.448058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.447321    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.455815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.449672    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.458745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.458720    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.503741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.503710    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwvr\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.503882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.503854    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.544756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.544727    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:51.562926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.562907    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:51.647410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.647385    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\") pod \"f542f4b0-039c-40e0-802c-e67f5f474704\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") "
Nov 03 21:07:51.647538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.647520    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\" (UniqueName: \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\") pod \"f542f4b0-039c-40e0-802c-e67f5f474704\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") "
Nov 03 21:07:51.647622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.647607    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\") pod \"f542f4b0-039c-40e0-802c-e67f5f474704\" (UID: \"f542f4b0-039c-40e0-802c-e67f5f474704\") "
Nov 03 21:07:51.648016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.647987    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca" pod "f542f4b0-039c-40e0-802c-e67f5f474704" (UID: "f542f4b0-039c-40e0-802c-e67f5f474704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:51.648142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.648081    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:51.649494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.649471    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca" pod "f542f4b0-039c-40e0-802c-e67f5f474704" (UID: "f542f4b0-039c-40e0-802c-e67f5f474704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:51.649693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.649668    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj" pod "f542f4b0-039c-40e0-802c-e67f5f474704" (UID: "f542f4b0-039c-40e0-802c-e67f5f474704"). InnerVolumeSpecName "kube-api-access-9cpmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:51.749756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.749689    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f542f4b0-039c-40e0-802c-e67f5f474704-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:51.749756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.749714    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpmj\" (UniqueName: \"kubernetes.io/projected/f542f4b0-039c-40e0-802c-e67f5f474704-kube-api-access-9cpmj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:51.962958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.962918    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"]
Nov 03 21:07:51.963972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.963939    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f542f4b0-039c-40e0-802c-e67f5f474704" containerName="revoke-keys"
Nov 03 21:07:51.963972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.963966    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f542f4b0-039c-40e0-802c-e67f5f474704" containerName="revoke-keys"
Nov 03 21:07:51.964157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.964114    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f542f4b0-039c-40e0-802c-e67f5f474704" containerName="revoke-keys"
Nov 03 21:07:51.964157 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:51.964137    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5e727e_11a6_43cd_90b3_b8fa803b0aca.slice/crio-e6d8971c0586287c8d7f9dfde529335b0faf3791f6aed5e91a966ced833749ca WatchSource:0}: Error finding container e6d8971c0586287c8d7f9dfde529335b0faf3791f6aed5e91a966ced833749ca: Status 404 returned error can't find the container with id e6d8971c0586287c8d7f9dfde529335b0faf3791f6aed5e91a966ced833749ca
Nov 03 21:07:51.967236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.967197    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:51.967343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.967261    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:51.967343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.967299    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:51.968103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.968087    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:07:51.968198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:51.968184    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.022054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.022025    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"]
Nov 03 21:07:52.052484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.052451    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.052629 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.052497    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrv8\" (UniqueName: \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.052724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.052672    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.153427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.153395    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.153427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.153429    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrv8\" (UniqueName: \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.153616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.153490    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.154110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.154091    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.155715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.155697    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.210829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.210803    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrv8\" (UniqueName: \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\") pod \"maas-api-revoke-sub-5a9c5b73-tslcg\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.289487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.289297    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:52.439441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.439409    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"
Nov 03 21:07:52.439441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.439435    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd" event={"ID":"f542f4b0-039c-40e0-802c-e67f5f474704","Type":"ContainerDied","Data":"d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e"}
Nov 03 21:07:52.439685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.439489    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8199601339b9b81c1f407b98b897188aefd5c3528af6f8dac5da1ff8162b14e"
Nov 03 21:07:52.442614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.442574    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" event={"ID":"fb5e727e-11a6-43cd-90b3-b8fa803b0aca","Type":"ContainerStarted","Data":"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"}
Nov 03 21:07:52.442806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.442655    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" event={"ID":"fb5e727e-11a6-43cd-90b3-b8fa803b0aca","Type":"ContainerStarted","Data":"e6d8971c0586287c8d7f9dfde529335b0faf3791f6aed5e91a966ced833749ca"}
Nov 03 21:07:52.545600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.545520    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:52.578660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.575786    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podStartSLOduration=1.575766197 podStartE2EDuration="1.575766197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:52.564442089 +0000 UTC m=+1174.195755567" watchObservedRunningTime="2026-11-03 21:07:52.575766197 +0000 UTC m=+1174.207079677"
Nov 03 21:07:52.678988 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:07:52.678953    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867e50a4_9417_4e14_bca7_61d4ee99b280.slice/crio-d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66 WatchSource:0}: Error finding container d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66: Status 404 returned error can't find the container with id d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66
Nov 03 21:07:52.679376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:52.679350    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"]
Nov 03 21:07:53.450654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:53.450598    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg" event={"ID":"867e50a4-9417-4e14-bca7-61d4ee99b280","Type":"ContainerStarted","Data":"b77cf22278b4f6ef5f89e8d8e8f803547510757a3c151d7b543f4cf7e63348c7"}
Nov 03 21:07:53.450867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:53.450684    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg" event={"ID":"867e50a4-9417-4e14-bca7-61d4ee99b280","Type":"ContainerStarted","Data":"d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66"}
Nov 03 21:07:53.546411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:53.546377    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:53.546797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:53.546428    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:53.579013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:53.578953    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg" podStartSLOduration=2.578933683 podStartE2EDuration="2.578933683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:53.562883392 +0000 UTC m=+1175.194196871" watchObservedRunningTime="2026-11-03 21:07:53.578933683 +0000 UTC m=+1175.210247161"
Nov 03 21:07:54.490013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:54.489975    5533 generic.go:356] "Generic (PLEG): container finished" podID="867e50a4-9417-4e14-bca7-61d4ee99b280" containerID="b77cf22278b4f6ef5f89e8d8e8f803547510757a3c151d7b543f4cf7e63348c7" exitCode=0
Nov 03 21:07:54.493346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:54.493320    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg" event={"ID":"867e50a4-9417-4e14-bca7-61d4ee99b280","Type":"ContainerDied","Data":"b77cf22278b4f6ef5f89e8d8e8f803547510757a3c151d7b543f4cf7e63348c7"}
Nov 03 21:07:54.545402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:54.545376    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:54.545505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:54.545418    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:55.545917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.545881    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:55.546294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.545947    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:55.685773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.685747    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:55.714465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.714441    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\" (UniqueName: \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\") pod \"867e50a4-9417-4e14-bca7-61d4ee99b280\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") "
Nov 03 21:07:55.714601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.714480    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\") pod \"867e50a4-9417-4e14-bca7-61d4ee99b280\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") "
Nov 03 21:07:55.714601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.714526    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\") pod \"867e50a4-9417-4e14-bca7-61d4ee99b280\" (UID: \"867e50a4-9417-4e14-bca7-61d4ee99b280\") "
Nov 03 21:07:55.714949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.714921    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca" pod "867e50a4-9417-4e14-bca7-61d4ee99b280" (UID: "867e50a4-9417-4e14-bca7-61d4ee99b280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:07:55.716490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.716469    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca" pod "867e50a4-9417-4e14-bca7-61d4ee99b280" (UID: "867e50a4-9417-4e14-bca7-61d4ee99b280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:55.716570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.716534    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8" pod "867e50a4-9417-4e14-bca7-61d4ee99b280" (UID: "867e50a4-9417-4e14-bca7-61d4ee99b280"). InnerVolumeSpecName "kube-api-access-gmrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:55.815995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.815924    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrv8\" (UniqueName: \"kubernetes.io/projected/867e50a4-9417-4e14-bca7-61d4ee99b280-kube-api-access-gmrv8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:55.815995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.815951    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:55.815995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:55.815964    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867e50a4-9417-4e14-bca7-61d4ee99b280-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:56.504546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:56.504517    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"
Nov 03 21:07:56.504717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:56.504550    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg" event={"ID":"867e50a4-9417-4e14-bca7-61d4ee99b280","Type":"ContainerDied","Data":"d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66"}
Nov 03 21:07:56.504717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:56.504596    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b59ba0922b0b6545a551e9eb9a78a25c68b4870cdebdbb43861dc028679e66"
Nov 03 21:07:56.545927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:56.545897    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:56.546271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:56.545953    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:57.545592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:57.545556    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:57.545750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:57.545608    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:58.545444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:58.545413    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 21:07:58.545830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:58.545466    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 21:07:59.549469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:59.549389    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:59.549899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:59.549821    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:07:59.550543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:07:59.550523    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:08:01.901472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:01.901438    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:08:05.134103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.134065    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"]
Nov 03 21:08:05.134600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.134579    5533 cpu_manager.go:414] "containerMap: removing container" podUID="867e50a4-9417-4e14-bca7-61d4ee99b280" containerName="revoke-keys"
Nov 03 21:08:05.134600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.134596    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867e50a4-9417-4e14-bca7-61d4ee99b280" containerName="revoke-keys"
Nov 03 21:08:05.134744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.134733    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="867e50a4-9417-4e14-bca7-61d4ee99b280" containerName="revoke-keys"
Nov 03 21:08:05.139596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.139573    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.193377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.193354    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"]
Nov 03 21:08:05.235190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.235161    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:05.235335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.235195    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:05.242843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.242819    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt452\" (UniqueName: \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.242943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.242904    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.243011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.242990    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.344648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.344607    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.344820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.344802    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt452\" (UniqueName: \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.344896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.344841    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.345266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.345244    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.347208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.347188    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.402384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.402328    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt452\" (UniqueName: \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\") pod \"maas-api-revoke-sub-8f123d07-w2wlw\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.454943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.454925    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:05.475315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.475287    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:08:05.479920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.479897    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.540555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.540470    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:08:05.547662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547612    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547687    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547709    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547761    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547801    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547831    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn64j\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547858    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547878    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.547944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.547909    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649401    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649461    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649503    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649520    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649577    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649615    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649683    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn64j\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649729    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649748    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.649943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649907    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.650086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.649949    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.650086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.650028    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.650234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.650216    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.650602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.650580    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.651775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.651754    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.652102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.652081    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.716829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.716803    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.716956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.716931    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn64j\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.796173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.796144    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:05.865569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:05.865543    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"]
Nov 03 21:08:06.168464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.168437    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:08:06.168817 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:06.168779    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3758ae8_f613_46fd_9e22_8497c43c9e1a.slice/crio-920b529bf3f86e2f9a64a5dbf46a0283242486e0f222504919a8ba810f83d90c WatchSource:0}: Error finding container 920b529bf3f86e2f9a64a5dbf46a0283242486e0f222504919a8ba810f83d90c: Status 404 returned error can't find the container with id 920b529bf3f86e2f9a64a5dbf46a0283242486e0f222504919a8ba810f83d90c
Nov 03 21:08:06.171500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.171468    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:06.171600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.171530    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:06.171600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.171560    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:06.580175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.580134    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw" event={"ID":"6917e9e9-01fc-41b7-b175-3bae5636ab29","Type":"ContainerStarted","Data":"baaf8deaf613e791828ace0b1aa9ce612c6968865d89cb3184ac03c3749f3b42"}
Nov 03 21:08:06.580360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.580211    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw" event={"ID":"6917e9e9-01fc-41b7-b175-3bae5636ab29","Type":"ContainerStarted","Data":"3354ea9a8f0dc50a64944410c3ab4a7acf66a50e8626f4c50904512ead8a34be"}
Nov 03 21:08:06.582056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.582022    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" event={"ID":"f3758ae8-f613-46fd-9e22-8497c43c9e1a","Type":"ContainerStarted","Data":"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"}
Nov 03 21:08:06.582175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.582076    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" event={"ID":"f3758ae8-f613-46fd-9e22-8497c43c9e1a","Type":"ContainerStarted","Data":"920b529bf3f86e2f9a64a5dbf46a0283242486e0f222504919a8ba810f83d90c"}
Nov 03 21:08:06.691832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.691778    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw" podStartSLOduration=1.691763614 podStartE2EDuration="1.691763614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:06.688773606 +0000 UTC m=+1188.320087084" watchObservedRunningTime="2026-11-03 21:08:06.691763614 +0000 UTC m=+1188.323077092"
Nov 03 21:08:06.797187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.797153    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:06.804626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:06.804577    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podStartSLOduration=1.804563906 podStartE2EDuration="1.804563906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:06.802844086 +0000 UTC m=+1188.434157563" watchObservedRunningTime="2026-11-03 21:08:06.804563906 +0000 UTC m=+1188.435877383"
Nov 03 21:08:07.591109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:07.591074    5533 generic.go:356] "Generic (PLEG): container finished" podID="6917e9e9-01fc-41b7-b175-3bae5636ab29" containerID="baaf8deaf613e791828ace0b1aa9ce612c6968865d89cb3184ac03c3749f3b42" exitCode=0
Nov 03 21:08:07.591524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:07.591194    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw" event={"ID":"6917e9e9-01fc-41b7-b175-3bae5636ab29","Type":"ContainerDied","Data":"baaf8deaf613e791828ace0b1aa9ce612c6968865d89cb3184ac03c3749f3b42"}
Nov 03 21:08:07.798818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:07.798783    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:07.798962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:07.798834    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:08.280572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.280535    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"]
Nov 03 21:08:08.284500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.284480    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.371657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.350591    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"]
Nov 03 21:08:08.395346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.395311    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.395507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.395366    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.395709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.395676    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkx7p\" (UniqueName: \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.497322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.497290    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.497488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.497411    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkx7p\" (UniqueName: \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.497563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.497494    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.498128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.498106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.499630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.499604    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.573694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.572659    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkx7p\" (UniqueName: \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\") pod \"maas-api-revoke-sub-8511881a-pqhpw\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.599373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.598402    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:08.797615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.797584    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:08.797747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.797664    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:08.798329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.798179    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:08.902828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.902748    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\") pod \"6917e9e9-01fc-41b7-b175-3bae5636ab29\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") "
Nov 03 21:08:08.902978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.902858    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\" (UniqueName: \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\") pod \"6917e9e9-01fc-41b7-b175-3bae5636ab29\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") "
Nov 03 21:08:08.902978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.902894    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\") pod \"6917e9e9-01fc-41b7-b175-3bae5636ab29\" (UID: \"6917e9e9-01fc-41b7-b175-3bae5636ab29\") "
Nov 03 21:08:08.903181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.903145    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca" pod "6917e9e9-01fc-41b7-b175-3bae5636ab29" (UID: "6917e9e9-01fc-41b7-b175-3bae5636ab29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:08.903460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.903429    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:08.905041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.905016    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca" pod "6917e9e9-01fc-41b7-b175-3bae5636ab29" (UID: "6917e9e9-01fc-41b7-b175-3bae5636ab29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:08.905041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:08.905025    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452" pod "6917e9e9-01fc-41b7-b175-3bae5636ab29" (UID: "6917e9e9-01fc-41b7-b175-3bae5636ab29"). InnerVolumeSpecName "kube-api-access-bt452". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:09.004326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.004291    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt452\" (UniqueName: \"kubernetes.io/projected/6917e9e9-01fc-41b7-b175-3bae5636ab29-kube-api-access-bt452\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:09.004326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.004329    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917e9e9-01fc-41b7-b175-3bae5636ab29-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:09.018605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.018576    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"]
Nov 03 21:08:09.019892 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:09.019859    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ece2f3_f999_4ccb_bc82_b9e722450c49.slice/crio-08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938 WatchSource:0}: Error finding container 08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938: Status 404 returned error can't find the container with id 08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938
Nov 03 21:08:09.606525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.606435    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"
Nov 03 21:08:09.606525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.606489    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw" event={"ID":"6917e9e9-01fc-41b7-b175-3bae5636ab29","Type":"ContainerDied","Data":"3354ea9a8f0dc50a64944410c3ab4a7acf66a50e8626f4c50904512ead8a34be"}
Nov 03 21:08:09.607080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.606551    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3354ea9a8f0dc50a64944410c3ab4a7acf66a50e8626f4c50904512ead8a34be"
Nov 03 21:08:09.608592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.608570    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw" event={"ID":"a2ece2f3-f999-4ccb-bc82-b9e722450c49","Type":"ContainerStarted","Data":"c645d41a4c81a96d296e43869fe8fc866272f9865983c3e8b8777698148c22bf"}
Nov 03 21:08:09.608723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.608608    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw" event={"ID":"a2ece2f3-f999-4ccb-bc82-b9e722450c49","Type":"ContainerStarted","Data":"08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938"}
Nov 03 21:08:09.728646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.728585    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw" podStartSLOduration=1.728571978 podStartE2EDuration="1.728571978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:09.725684982 +0000 UTC m=+1191.356998460" watchObservedRunningTime="2026-11-03 21:08:09.728571978 +0000 UTC m=+1191.359885481"
Nov 03 21:08:09.797027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.796994    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:09.797180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:09.797046    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:10.616608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:10.616574    5533 generic.go:356] "Generic (PLEG): container finished" podID="a2ece2f3-f999-4ccb-bc82-b9e722450c49" containerID="c645d41a4c81a96d296e43869fe8fc866272f9865983c3e8b8777698148c22bf" exitCode=0
Nov 03 21:08:10.617094 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:10.616669    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw" event={"ID":"a2ece2f3-f999-4ccb-bc82-b9e722450c49","Type":"ContainerDied","Data":"c645d41a4c81a96d296e43869fe8fc866272f9865983c3e8b8777698148c22bf"}
Nov 03 21:08:10.797169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:10.797137    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:10.797450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:10.797204    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:11.796598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.796534    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:11.796598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.796588    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:11.821946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.821715    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:11.937184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.937156    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\") pod \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") "
Nov 03 21:08:11.937908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.937879    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\") pod \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") "
Nov 03 21:08:11.938134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.938114    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\" (UniqueName: \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\") pod \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\" (UID: \"a2ece2f3-f999-4ccb-bc82-b9e722450c49\") "
Nov 03 21:08:11.938300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.937949    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca" pod "a2ece2f3-f999-4ccb-bc82-b9e722450c49" (UID: "a2ece2f3-f999-4ccb-bc82-b9e722450c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:11.938953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.938931    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:11.940245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.940221    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca" pod "a2ece2f3-f999-4ccb-bc82-b9e722450c49" (UID: "a2ece2f3-f999-4ccb-bc82-b9e722450c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:11.940626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:11.940605    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p" pod "a2ece2f3-f999-4ccb-bc82-b9e722450c49" (UID: "a2ece2f3-f999-4ccb-bc82-b9e722450c49"). InnerVolumeSpecName "kube-api-access-qkx7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:12.040662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.040606    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkx7p\" (UniqueName: \"kubernetes.io/projected/a2ece2f3-f999-4ccb-bc82-b9e722450c49-kube-api-access-qkx7p\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:12.040823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.040669    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ece2f3-f999-4ccb-bc82-b9e722450c49-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:12.632185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.632154    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"
Nov 03 21:08:12.632346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.632149    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw" event={"ID":"a2ece2f3-f999-4ccb-bc82-b9e722450c49","Type":"ContainerDied","Data":"08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938"}
Nov 03 21:08:12.632346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.632284    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08af37f49049013ade709351768ae653f3d8d38e787251eed50f13b927e41938"
Nov 03 21:08:12.796843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.796758    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 21:08:12.796843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.796824    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 21:08:12.850963 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.850931    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 21:08:12.851587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851565    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a2ece2f3-f999-4ccb-bc82-b9e722450c49" containerName="revoke-keys"
Nov 03 21:08:12.851587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851588    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ece2f3-f999-4ccb-bc82-b9e722450c49" containerName="revoke-keys"
Nov 03 21:08:12.851803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851625    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6917e9e9-01fc-41b7-b175-3bae5636ab29" containerName="revoke-keys"
Nov 03 21:08:12.851803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851646    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6917e9e9-01fc-41b7-b175-3bae5636ab29" containerName="revoke-keys"
Nov 03 21:08:12.851803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851784    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6917e9e9-01fc-41b7-b175-3bae5636ab29" containerName="revoke-keys"
Nov 03 21:08:12.851957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.851805    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ece2f3-f999-4ccb-bc82-b9e722450c49" containerName="revoke-keys"
Nov 03 21:08:12.855325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.855305    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.913144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.913117    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 21:08:12.949415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.949387    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-320f60\"/\"distinct-2-w3-320f60-kserve-self-signed-certs\""
Nov 03 21:08:12.952043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952014    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.952170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952056    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.952241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952213    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.952289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952273    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.952399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952384    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:12.952490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:12.952473    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7g9\" (UniqueName: \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.053815    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7g9\" (UniqueName: \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.053913    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.053942    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.054028    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.054073    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.054144    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.054625    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.055495    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.055895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.055606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.058178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.058154    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.059083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.059059    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.113557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.113528    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7g9\" (UniqueName: \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\") pod \"distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") " pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.168243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.168208    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:13.613387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.613355    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 21:08:13.614050 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:13.614014    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7df02540_271e_4f6e_bc1d_1778fc2f5f0e.slice/crio-caeb82419bbacace3f8df2dc3bda793ec70c1bde3a66c1a594b5023b5b505818 WatchSource:0}: Error finding container caeb82419bbacace3f8df2dc3bda793ec70c1bde3a66c1a594b5023b5b505818: Status 404 returned error can't find the container with id caeb82419bbacace3f8df2dc3bda793ec70c1bde3a66c1a594b5023b5b505818
Nov 03 21:08:13.639986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.639961    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" event={"ID":"7df02540-271e-4f6e-bc1d-1778fc2f5f0e","Type":"ContainerStarted","Data":"caeb82419bbacace3f8df2dc3bda793ec70c1bde3a66c1a594b5023b5b505818"}
Nov 03 21:08:13.802252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.802215    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:13.803071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.803045    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:13.803864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:13.803843    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:08:14.648062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:14.648024    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" event={"ID":"7df02540-271e-4f6e-bc1d-1778fc2f5f0e","Type":"ContainerStarted","Data":"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"}
Nov 03 21:08:14.767149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:14.767099    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" podStartSLOduration=2.767083818 podStartE2EDuration="2.767083818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:14.763083671 +0000 UTC m=+1196.394397149" watchObservedRunningTime="2026-11-03 21:08:14.767083818 +0000 UTC m=+1196.398397295"
Nov 03 21:08:15.169581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:15.169546    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:18.062305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.062271    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"]
Nov 03 21:08:18.109204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.109171    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79733355-gschr"]
Nov 03 21:08:18.493936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.493908    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9" path="/var/lib/kubelet/pods/b18cfa04-f452-44c2-96c2-c4e2d4ddf8a9/volumes"
Nov 03 21:08:18.644887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.644860    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:08:18.647096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.647066    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:08:18.656363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.656339    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:08:18.659161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:18.659140    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:08:19.009716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.009629    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"]
Nov 03 21:08:19.014819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.014797    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.065785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.065764    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:19.070459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.070434    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"]
Nov 03 21:08:19.086863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.086838    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2sm\" (UniqueName: \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.086959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.086892    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.087013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.086981    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.112355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.112330    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:19.188359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.188331    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2sm\" (UniqueName: \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.188512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.188385    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.188512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.188484    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.189270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.189243    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.191033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.191012    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.246242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.246209    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2sm\" (UniqueName: \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\") pod \"maas-api-revoke-sub-fba17051-d428z\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.261696 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:08:19.261620    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:08:19.327968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.327940    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:19.704364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.702269    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"]
Nov 03 21:08:19.792203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:19.792171    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z" event={"ID":"48055325-b2bf-4366-9ca9-6e193241f991","Type":"ContainerStarted","Data":"046b186d01dc28b5858aec10665877654214b1336cc3c0702db86bcc8a6711a9"}
Nov 03 21:08:20.801627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:20.801592    5533 generic.go:356] "Generic (PLEG): container finished" podID="48055325-b2bf-4366-9ca9-6e193241f991" containerID="b4bd181e57694416ac4285e692b5153d67d6a4887bffd072a0cdece938a36993" exitCode=0
Nov 03 21:08:20.802030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:20.801684    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z" event={"ID":"48055325-b2bf-4366-9ca9-6e193241f991","Type":"ContainerDied","Data":"b4bd181e57694416ac4285e692b5153d67d6a4887bffd072a0cdece938a36993"}
Nov 03 21:08:21.959218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:21.959192    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:22.021761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.021735    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\") pod \"48055325-b2bf-4366-9ca9-6e193241f991\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") "
Nov 03 21:08:22.021915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.021899    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\" (UniqueName: \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\") pod \"48055325-b2bf-4366-9ca9-6e193241f991\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") "
Nov 03 21:08:22.021982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.021931    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\") pod \"48055325-b2bf-4366-9ca9-6e193241f991\" (UID: \"48055325-b2bf-4366-9ca9-6e193241f991\") "
Nov 03 21:08:22.022300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.022277    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca" pod "48055325-b2bf-4366-9ca9-6e193241f991" (UID: "48055325-b2bf-4366-9ca9-6e193241f991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:22.022436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.022419    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:22.023797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.023776    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca" pod "48055325-b2bf-4366-9ca9-6e193241f991" (UID: "48055325-b2bf-4366-9ca9-6e193241f991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:22.023894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.023876    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm" pod "48055325-b2bf-4366-9ca9-6e193241f991" (UID: "48055325-b2bf-4366-9ca9-6e193241f991"). InnerVolumeSpecName "kube-api-access-tt2sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:22.109034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.108957    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"]
Nov 03 21:08:22.109457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.109439    5533 cpu_manager.go:414] "containerMap: removing container" podUID="48055325-b2bf-4366-9ca9-6e193241f991" containerName="revoke-keys"
Nov 03 21:08:22.109457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.109456    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48055325-b2bf-4366-9ca9-6e193241f991" containerName="revoke-keys"
Nov 03 21:08:22.109622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.109567    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="48055325-b2bf-4366-9ca9-6e193241f991" containerName="revoke-keys"
Nov 03 21:08:22.113396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.113374    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.123568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.123548    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48055325-b2bf-4366-9ca9-6e193241f991-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:22.123568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.123569    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2sm\" (UniqueName: \"kubernetes.io/projected/48055325-b2bf-4366-9ca9-6e193241f991-kube-api-access-tt2sm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:22.168210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.168177    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"]
Nov 03 21:08:22.224360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.224331    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vck67\" (UniqueName: \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.224474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.224438    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.224526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.224489    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.326196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.326158    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vck67\" (UniqueName: \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.326395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.326278    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.326395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.326312    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.327027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.327007    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.328849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.328817    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.383601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.383527    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vck67\" (UniqueName: \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\") pod \"maas-api-revoke-sub-9429373e-6vddm\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.431109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.431085    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:22.798904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.798879    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"]
Nov 03 21:08:22.800162 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:22.800134    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c452b2_814d_4045_86b2_5253c3ff772e.slice/crio-590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550 WatchSource:0}: Error finding container 590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550: Status 404 returned error can't find the container with id 590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550
Nov 03 21:08:22.822982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.822950    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm" event={"ID":"06c452b2-814d-4045-86b2-5253c3ff772e","Type":"ContainerStarted","Data":"590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550"}
Nov 03 21:08:22.824497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.824478    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"
Nov 03 21:08:22.824497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.824486    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z" event={"ID":"48055325-b2bf-4366-9ca9-6e193241f991","Type":"ContainerDied","Data":"046b186d01dc28b5858aec10665877654214b1336cc3c0702db86bcc8a6711a9"}
Nov 03 21:08:22.824618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:22.824533    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="046b186d01dc28b5858aec10665877654214b1336cc3c0702db86bcc8a6711a9"
Nov 03 21:08:23.169006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:23.168977    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:23.181688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:23.181665    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:23.832586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:23.832503    5533 generic.go:356] "Generic (PLEG): container finished" podID="06c452b2-814d-4045-86b2-5253c3ff772e" containerID="ecc072b07eae4899cb7d426708e5484ca1de0ce5633902c6a3f115568f14e16a" exitCode=0
Nov 03 21:08:23.832782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:23.832587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm" event={"ID":"06c452b2-814d-4045-86b2-5253c3ff772e","Type":"ContainerDied","Data":"ecc072b07eae4899cb7d426708e5484ca1de0ce5633902c6a3f115568f14e16a"}
Nov 03 21:08:23.844746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:23.844725    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:08:24.997621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:24.997588    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:25.055066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.055036    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\") pod \"06c452b2-814d-4045-86b2-5253c3ff772e\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") "
Nov 03 21:08:25.055214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.055138    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\" (UniqueName: \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\") pod \"06c452b2-814d-4045-86b2-5253c3ff772e\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") "
Nov 03 21:08:25.055214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.055204    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\") pod \"06c452b2-814d-4045-86b2-5253c3ff772e\" (UID: \"06c452b2-814d-4045-86b2-5253c3ff772e\") "
Nov 03 21:08:25.055430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.055400    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca" pod "06c452b2-814d-4045-86b2-5253c3ff772e" (UID: "06c452b2-814d-4045-86b2-5253c3ff772e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:25.055625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.055608    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.057786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.057750    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca" pod "06c452b2-814d-4045-86b2-5253c3ff772e" (UID: "06c452b2-814d-4045-86b2-5253c3ff772e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:25.057786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.057766    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67" pod "06c452b2-814d-4045-86b2-5253c3ff772e" (UID: "06c452b2-814d-4045-86b2-5253c3ff772e"). InnerVolumeSpecName "kube-api-access-vck67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:25.156658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.156570    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vck67\" (UniqueName: \"kubernetes.io/projected/06c452b2-814d-4045-86b2-5253c3ff772e-kube-api-access-vck67\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.156658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.156607    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c452b2-814d-4045-86b2-5253c3ff772e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.848235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.848201    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm" event={"ID":"06c452b2-814d-4045-86b2-5253c3ff772e","Type":"ContainerDied","Data":"590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550"}
Nov 03 21:08:25.848235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.848225    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"
Nov 03 21:08:25.848429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:25.848250    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590d9f54ad8315f565095ea3a94401adb37789ba96e6773cd90480c9879c1550"
Nov 03 21:08:26.883354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:26.883319    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:08:29.235658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.232420    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:29.251660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.240712    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podUID="3b4da839-0fd8-4d9e-81a2-371496377bba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:29.538817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.538716    5533 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 21:08:29.594919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.594882    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:29.760659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.759710    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:29.986944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:29.986887    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:30.118741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:30.118701    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 21:08:30.243288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:30.243169    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podUID="3b4da839-0fd8-4d9e-81a2-371496377bba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:30.243760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:30.243531    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 21:08:30.479624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:30.479592    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 21:08:32.889475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.889443    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"]
Nov 03 21:08:32.889956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.889942    5533 cpu_manager.go:414] "containerMap: removing container" podUID="06c452b2-814d-4045-86b2-5253c3ff772e" containerName="revoke-keys"
Nov 03 21:08:32.890010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.889957    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c452b2-814d-4045-86b2-5253c3ff772e" containerName="revoke-keys"
Nov 03 21:08:32.890051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.890041    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c452b2-814d-4045-86b2-5253c3ff772e" containerName="revoke-keys"
Nov 03 21:08:32.894829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.894808    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:32.975351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.975326    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:32.975482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.975458    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxtr\" (UniqueName: \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:32.975575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.975558    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:32.990853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.990817    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:32.990853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.990832    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:32.996919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:32.996897    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"]
Nov 03 21:08:33.076826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.076800    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxtr\" (UniqueName: \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.076939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.076873    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.076939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.076930    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.077522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.077502    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.079378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.079353    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.134908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.134884    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxtr\" (UniqueName: \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\") pod \"maas-api-revoke-sub-266267b1-c76b4\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.207819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.207798    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:33.576049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:33.576011    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"]
Nov 03 21:08:33.576577 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:33.576550    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ebe29d_2c4c_44ff_9904_992d297f1f89.slice/crio-f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903 WatchSource:0}: Error finding container f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903: Status 404 returned error can't find the container with id f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903
Nov 03 21:08:34.065345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:34.065306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4" event={"ID":"36ebe29d-2c4c-44ff-9904-992d297f1f89","Type":"ContainerStarted","Data":"8dc15af8b544dad2ba8f32adc0a40bdc37afc8d38ec0cd12f46afbb79b33f20e"}
Nov 03 21:08:34.065796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:34.065370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4" event={"ID":"36ebe29d-2c4c-44ff-9904-992d297f1f89","Type":"ContainerStarted","Data":"f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903"}
Nov 03 21:08:34.175667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:34.175610    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4" podStartSLOduration=2.175597308 podStartE2EDuration="2.175597308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:34.173767985 +0000 UTC m=+1215.805081459" watchObservedRunningTime="2026-11-03 21:08:34.175597308 +0000 UTC m=+1215.806910784"
Nov 03 21:08:35.074101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:35.074065    5533 generic.go:356] "Generic (PLEG): container finished" podID="36ebe29d-2c4c-44ff-9904-992d297f1f89" containerID="8dc15af8b544dad2ba8f32adc0a40bdc37afc8d38ec0cd12f46afbb79b33f20e" exitCode=0
Nov 03 21:08:35.074516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:35.074150    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4" event={"ID":"36ebe29d-2c4c-44ff-9904-992d297f1f89","Type":"ContainerDied","Data":"8dc15af8b544dad2ba8f32adc0a40bdc37afc8d38ec0cd12f46afbb79b33f20e"}
Nov 03 21:08:36.046983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.046948    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"]
Nov 03 21:08:36.099465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.099433    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ada009b-6xcbw"]
Nov 03 21:08:36.236512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.236489    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:36.237525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.237503    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:08:36.238064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.238049    5533 cpu_manager.go:414] "containerMap: removing container" podUID="36ebe29d-2c4c-44ff-9904-992d297f1f89" containerName="revoke-keys"
Nov 03 21:08:36.238126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.238065    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ebe29d-2c4c-44ff-9904-992d297f1f89" containerName="revoke-keys"
Nov 03 21:08:36.238174 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.238164    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ebe29d-2c4c-44ff-9904-992d297f1f89" containerName="revoke-keys"
Nov 03 21:08:36.241342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.241324    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.297929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.297846    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:08:36.303214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.303188    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.308528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.308498    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\" (UniqueName: \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\") pod \"36ebe29d-2c4c-44ff-9904-992d297f1f89\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") "
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.308580    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\") pod \"36ebe29d-2c4c-44ff-9904-992d297f1f89\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") "
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.308630    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\") pod \"36ebe29d-2c4c-44ff-9904-992d297f1f89\" (UID: \"36ebe29d-2c4c-44ff-9904-992d297f1f89\") "
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.308930    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.308970    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309008    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309047    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309114    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309219    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktsc\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309360    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.309932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.309905    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca" pod "36ebe29d-2c4c-44ff-9904-992d297f1f89" (UID: "36ebe29d-2c4c-44ff-9904-992d297f1f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:36.311999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.311973    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca" pod "36ebe29d-2c4c-44ff-9904-992d297f1f89" (UID: "36ebe29d-2c4c-44ff-9904-992d297f1f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:36.312070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.312000    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr" pod "36ebe29d-2c4c-44ff-9904-992d297f1f89" (UID: "36ebe29d-2c4c-44ff-9904-992d297f1f89"). InnerVolumeSpecName "kube-api-access-lwxtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:36.326949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.326923    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:08:36.342014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.341988    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:08:36.410223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410188    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410237    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410278    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410315    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410374    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.410621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410404    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.410621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410442    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410500    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410548    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktsc\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410579    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.410936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410625    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.410995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.410934    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411045 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411014    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411261    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411303    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411307    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411384    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411424    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411522    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411573    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdxw\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411707    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411748    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.411807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411752    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.411915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411872    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwxtr\" (UniqueName: \"kubernetes.io/projected/36ebe29d-2c4c-44ff-9904-992d297f1f89-kube-api-access-lwxtr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.411915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411893    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.411915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.411907    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ebe29d-2c4c-44ff-9904-992d297f1f89-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:36.413067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.413041    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.414020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.413999    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.471579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.471552    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.471727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.471703    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktsc\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.496478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.496445    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddca59f1-e996-466a-8110-c7c113ff9ad7" path="/var/lib/kubelet/pods/ddca59f1-e996-466a-8110-c7c113ff9ad7/volumes"
Nov 03 21:08:36.512661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512609    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.512796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512664    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.512796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512692    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.512796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512770    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.512942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512811    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdxw\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.512838    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513382    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513339    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513414    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513113    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513506    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513528    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.513839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.513703    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.515780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.515756    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.516023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.516003    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.560836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.560780    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:36.571278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.571250    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.571372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.571334    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdxw\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.645066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.645031    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:36.988126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.988013    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:08:36.988496 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:36.988466    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2032c52_efe2_4944_91c8_f21325df3149.slice/crio-d1746d212b89272c3fa83182b8cf7b8180e8b70b42e58837d8680ff0663c14c2 WatchSource:0}: Error finding container d1746d212b89272c3fa83182b8cf7b8180e8b70b42e58837d8680ff0663c14c2: Status 404 returned error can't find the container with id d1746d212b89272c3fa83182b8cf7b8180e8b70b42e58837d8680ff0663c14c2
Nov 03 21:08:36.990791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.990760    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:36.990863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.990820    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:36.990863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:36.990848    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:37.038794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.038767    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:08:37.040117 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:37.040088    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8472e6eb_de5e_494a_a0f7_f1d98d326d9b.slice/crio-49d11e5880c857eb197d1ed85cf724ae421e7206361e90f35838ffd9c9e20187 WatchSource:0}: Error finding container 49d11e5880c857eb197d1ed85cf724ae421e7206361e90f35838ffd9c9e20187: Status 404 returned error can't find the container with id 49d11e5880c857eb197d1ed85cf724ae421e7206361e90f35838ffd9c9e20187
Nov 03 21:08:37.043066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.043030    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:37.043129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.043117    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:37.043181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.043163    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:37.089514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.089484    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4" event={"ID":"36ebe29d-2c4c-44ff-9904-992d297f1f89","Type":"ContainerDied","Data":"f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903"}
Nov 03 21:08:37.089617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.089543    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46701bf1013a8952185118db290ae57f164c6f229e5d6f1fdd77a6f13e18903"
Nov 03 21:08:37.089617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.089500    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"
Nov 03 21:08:37.092901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.092702    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" event={"ID":"8472e6eb-de5e-494a-a0f7-f1d98d326d9b","Type":"ContainerStarted","Data":"49d11e5880c857eb197d1ed85cf724ae421e7206361e90f35838ffd9c9e20187"}
Nov 03 21:08:37.094669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.094631    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" event={"ID":"e2032c52-efe2-4944-91c8-f21325df3149","Type":"ContainerStarted","Data":"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"}
Nov 03 21:08:37.094761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.094682    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" event={"ID":"e2032c52-efe2-4944-91c8-f21325df3149","Type":"ContainerStarted","Data":"d1746d212b89272c3fa83182b8cf7b8180e8b70b42e58837d8680ff0663c14c2"}
Nov 03 21:08:37.261313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.261204    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podStartSLOduration=1.26119012 podStartE2EDuration="1.26119012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:37.259932845 +0000 UTC m=+1218.891246331" watchObservedRunningTime="2026-11-03 21:08:37.26119012 +0000 UTC m=+1218.892503649"
Nov 03 21:08:37.349864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.349831    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"]
Nov 03 21:08:37.354837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.354810    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.404507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.404472    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:37.404698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.404614    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:37.409872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.409608    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"]
Nov 03 21:08:37.425772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.425733    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xks\" (UniqueName: \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.425935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.425899    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.426023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.426000    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.527187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.527110    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.527367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.527264    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xks\" (UniqueName: \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.527367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.527343    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.527858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.527837    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.529609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.529586    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.561843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.561817    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:37.585176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.585148    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xks\" (UniqueName: \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\") pod \"maas-api-revoke-sub-75eae842-wmtl4\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:37.674099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:37.674074    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:38.076659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.063918    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"]
Nov 03 21:08:38.107248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.107211    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4" event={"ID":"3025edb8-641e-4581-b9b1-4205b9c857f9","Type":"ContainerStarted","Data":"2e951210a8738769df3f708c4081a030c56a99fef82755ea38453d1bee455d57"}
Nov 03 21:08:38.109279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.109253    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" event={"ID":"8472e6eb-de5e-494a-a0f7-f1d98d326d9b","Type":"ContainerStarted","Data":"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"}
Nov 03 21:08:38.241041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.240989    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podStartSLOduration=2.240976513 podStartE2EDuration="2.240976513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:38.238066209 +0000 UTC m=+1219.869379683" watchObservedRunningTime="2026-11-03 21:08:38.240976513 +0000 UTC m=+1219.872289990"
Nov 03 21:08:38.581667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.580832    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:38.581667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.580927    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:38.646150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.646125    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:38.647646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.647601    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:38.647732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:38.647666    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:39.126368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.126335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4" event={"ID":"3025edb8-641e-4581-b9b1-4205b9c857f9","Type":"ContainerStarted","Data":"2dbd0fc354054bc08392a6afe00a17dba799f8aa534d1430626b2ba127c60a98"}
Nov 03 21:08:39.242293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.242241    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4" podStartSLOduration=2.242226859 podStartE2EDuration="2.242226859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:39.239105364 +0000 UTC m=+1220.870418836" watchObservedRunningTime="2026-11-03 21:08:39.242226859 +0000 UTC m=+1220.873540357"
Nov 03 21:08:39.567002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.566965    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:39.567275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.567244    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:39.646250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.646220    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:39.646714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:39.646690    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:40.134294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.134243    5533 generic.go:356] "Generic (PLEG): container finished" podID="3025edb8-641e-4581-b9b1-4205b9c857f9" containerID="2dbd0fc354054bc08392a6afe00a17dba799f8aa534d1430626b2ba127c60a98" exitCode=0
Nov 03 21:08:40.134465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.134339    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4" event={"ID":"3025edb8-641e-4581-b9b1-4205b9c857f9","Type":"ContainerDied","Data":"2dbd0fc354054bc08392a6afe00a17dba799f8aa534d1430626b2ba127c60a98"}
Nov 03 21:08:40.567659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.565961    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:40.567659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.566020    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:40.645595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.645565    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:40.645766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:40.645616    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:41.465998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.456818    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:41.529254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.529220    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\" (UniqueName: \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\") pod \"3025edb8-641e-4581-b9b1-4205b9c857f9\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") "
Nov 03 21:08:41.536001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.535973    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\") pod \"3025edb8-641e-4581-b9b1-4205b9c857f9\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") "
Nov 03 21:08:41.536149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.536018    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\") pod \"3025edb8-641e-4581-b9b1-4205b9c857f9\" (UID: \"3025edb8-641e-4581-b9b1-4205b9c857f9\") "
Nov 03 21:08:41.536535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.536510    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca" pod "3025edb8-641e-4581-b9b1-4205b9c857f9" (UID: "3025edb8-641e-4581-b9b1-4205b9c857f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:41.541502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.541477    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca" pod "3025edb8-641e-4581-b9b1-4205b9c857f9" (UID: "3025edb8-641e-4581-b9b1-4205b9c857f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:41.541674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.541628    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:41.541753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.541683    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3025edb8-641e-4581-b9b1-4205b9c857f9-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:41.541942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.541878    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks" pod "3025edb8-641e-4581-b9b1-4205b9c857f9" (UID: "3025edb8-641e-4581-b9b1-4205b9c857f9"). InnerVolumeSpecName "kube-api-access-z8xks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:41.573659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.571990    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:41.573659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.572054    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:41.643088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.642855    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xks\" (UniqueName: \"kubernetes.io/projected/3025edb8-641e-4581-b9b1-4205b9c857f9-kube-api-access-z8xks\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:41.646390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.646364    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:41.646521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:41.646415    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:42.149533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.149500    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4" event={"ID":"3025edb8-641e-4581-b9b1-4205b9c857f9","Type":"ContainerDied","Data":"2e951210a8738769df3f708c4081a030c56a99fef82755ea38453d1bee455d57"}
Nov 03 21:08:42.149800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.149572    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e951210a8738769df3f708c4081a030c56a99fef82755ea38453d1bee455d57"
Nov 03 21:08:42.149800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.149621    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"
Nov 03 21:08:42.570658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.563018    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:42.570658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.563090    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:42.645892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.645856    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:42.646063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:42.645914    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:43.564662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:43.564003    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:43.564662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:43.564082    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:43.649661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:43.646304    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:43.649661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:43.646371    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:44.572664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:44.563919    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 21:08:44.572664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:44.563977    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 21:08:44.650658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:44.646940    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 21:08:44.650658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:44.646999    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 21:08:45.005763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:45.005717    5533 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podUID="3b4da839-0fd8-4d9e-81a2-371496377bba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 21:08:45.203659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:45.203014    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:46.567492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:46.567394    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:46.567492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:46.567450    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:46.689813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:46.689768    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:46.690018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:46.689830    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:47.473529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.473478    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:47.474731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.474673    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:47.502187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.502112    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:08:47.610244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.609731    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:47.610244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.609782    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:47.676381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.674951    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:47.676381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.676222    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:47.692956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.692890    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:08:47.906462 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.906391    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"]
Nov 03 21:08:47.906983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.906965    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3025edb8-641e-4581-b9b1-4205b9c857f9" containerName="revoke-keys"
Nov 03 21:08:47.907105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.906984    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3025edb8-641e-4581-b9b1-4205b9c857f9" containerName="revoke-keys"
Nov 03 21:08:47.907105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.907103    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3025edb8-641e-4581-b9b1-4205b9c857f9" containerName="revoke-keys"
Nov 03 21:08:47.910863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.910831    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:47.960277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.960248    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:47.963946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.962996    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:47.965494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:47.965466    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"]
Nov 03 21:08:48.037765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.037739    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.037891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.037864    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5msf\" (UniqueName: \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.037935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.037900    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.138671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.138609    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5msf\" (UniqueName: \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.138820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.138691    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.138820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.138771    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.139431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.139405    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.141610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.141581    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.157956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.157897    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"]
Nov 03 21:08:48.162941 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.162921    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.196975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.196948    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5msf\" (UniqueName: \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\") pod \"maas-api-revoke-sub-fd957444-rwq4g\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.214341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.214318    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"]
Nov 03 21:08:48.224315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.224293    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:48.239833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.239809    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.239946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.239844    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92sxp\" (UniqueName: \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.240044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.240018    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.341380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.341351    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.341516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.341392    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92sxp\" (UniqueName: \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.341516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.341492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.342319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.342293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.343875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.343850    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.398841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.398813    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92sxp\" (UniqueName: \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\") pod \"maas-api-revoke-sub-5e84f5a8-bql8v\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.477206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.477178    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:48.524105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.523151    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:08:48.529751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.529720    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.625143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.624249    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:08:48.625755 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:48.625182    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a27319_b624_45e4_b778_0e5112e40432.slice/crio-f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99 WatchSource:0}: Error finding container f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99: Status 404 returned error can't find the container with id f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99
Nov 03 21:08:48.628008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.627983    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"]
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.645849    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.645956    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646140    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646205    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646264    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646317    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646359    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hnf\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646395    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.646814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.646464    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748070    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748114    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748174    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748194    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748284    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748323    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748369    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748402    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748428    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hnf\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748671    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748712    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748779    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.748904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.748819    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.749094 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.749073    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.750484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.750464    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.750484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.750473    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.805621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.805395    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.806597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.806568    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hnf\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\") pod \"e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") " pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.808918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.808894    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"]
Nov 03 21:08:48.814307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.814289    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:48.819278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.819257    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 21:08:48.823078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.823062    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.874058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.873182    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:48.882785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.882531    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"]
Nov 03 21:08:48.895728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.895621    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 21:08:48.904937 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:48.904762    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489547ee_81ea_49ee_bc50_63d063aa4e2c.slice/crio-b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c WatchSource:0}: Error finding container b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c: Status 404 returned error can't find the container with id b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c
Nov 03 21:08:48.905144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.905116    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"]
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955019    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955079    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955125    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955154    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955193    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxwc\" (UniqueName: \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955256    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgrw\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955313    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955342    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955404    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955429    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955509    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:48.957798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:48.955560    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056585    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056626    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056666    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056698    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxwc\" (UniqueName: \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.056821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056735    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgrw\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056786    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.056821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056816    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056871    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.057021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056896    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.056979    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.057169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057034    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057124    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057131    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057257    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057438    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057602    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.057934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057910    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.058019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.057964    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.059604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.059583    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.059727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.059708    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.059948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.059933    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.113850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.113831    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxwc\" (UniqueName: \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\") pod \"maas-api-revoke-sub-e21d8b9c-x6lq6\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.114423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.114406    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgrw\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.114474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.114447    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\") pod \"e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") " pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.129006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.128988    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:49.137071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.137055    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:49.264091 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:08:49.264045    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:08:49.274164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.274120    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:08:49.279416 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:49.279388    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110672cc_7b89_4860_a963_d39620d3486a.slice/crio-214b231d30527cb5927831fb91ff73ad59419666f5acce180ea80d201547e35e WatchSource:0}: Error finding container 214b231d30527cb5927831fb91ff73ad59419666f5acce180ea80d201547e35e: Status 404 returned error can't find the container with id 214b231d30527cb5927831fb91ff73ad59419666f5acce180ea80d201547e35e
Nov 03 21:08:49.284718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.282693    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.284718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.282836    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.284718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.282881    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.331917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.331621    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v" event={"ID":"489547ee-81ea-49ee-bc50-63d063aa4e2c","Type":"ContainerStarted","Data":"635a534ed255204f7cf57e80d2d2946e12b9a7dd37d056958bba55cd494bc91a"}
Nov 03 21:08:49.331917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.331697    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v" event={"ID":"489547ee-81ea-49ee-bc50-63d063aa4e2c","Type":"ContainerStarted","Data":"b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c"}
Nov 03 21:08:49.333248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.333215    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g" event={"ID":"82a27319-b624-45e4-b778-0e5112e40432","Type":"ContainerStarted","Data":"4f1782955156057a87a68bb1352c68d445175d3d538223e41de40919ad317f91"}
Nov 03 21:08:49.333332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.333273    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g" event={"ID":"82a27319-b624-45e4-b778-0e5112e40432","Type":"ContainerStarted","Data":"f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99"}
Nov 03 21:08:49.334848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.334821    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" event={"ID":"110672cc-7b89-4860-a963-d39620d3486a","Type":"ContainerStarted","Data":"214b231d30527cb5927831fb91ff73ad59419666f5acce180ea80d201547e35e"}
Nov 03 21:08:49.441587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.441534    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v" podStartSLOduration=1.44151856 podStartE2EDuration="1.44151856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:49.439906793 +0000 UTC m=+1231.071220273" watchObservedRunningTime="2026-11-03 21:08:49.44151856 +0000 UTC m=+1231.072832036"
Nov 03 21:08:49.522189 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:49.522085    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654fd73d_4cf1_41d1_b0a2_ee56728b1cfe.slice/crio-177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a WatchSource:0}: Error finding container 177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a: Status 404 returned error can't find the container with id 177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a
Nov 03 21:08:49.522317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.522195    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"]
Nov 03 21:08:49.527898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.527869    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 21:08:49.528455 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:08:49.528432    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c65baa_882c_4cce_b5c7_5754d1fbff81.slice/crio-16a6950fe06151e733d6829493ff52d3ab3ceb393f5e5e16d0bbbadb5c6348f4 WatchSource:0}: Error finding container 16a6950fe06151e733d6829493ff52d3ab3ceb393f5e5e16d0bbbadb5c6348f4: Status 404 returned error can't find the container with id 16a6950fe06151e733d6829493ff52d3ab3ceb393f5e5e16d0bbbadb5c6348f4
Nov 03 21:08:49.536059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.536026    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.536147 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.536108    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.536183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.536152    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:08:49.558669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:49.554733    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g" podStartSLOduration=2.554715057 podStartE2EDuration="2.554715057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:49.547526995 +0000 UTC m=+1231.178840475" watchObservedRunningTime="2026-11-03 21:08:49.554715057 +0000 UTC m=+1231.186028536"
Nov 03 21:08:50.049291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.049254    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"]
Nov 03 21:08:50.071657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.062986    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4523182d-zk4r5"]
Nov 03 21:08:50.358661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.355605    5533 generic.go:356] "Generic (PLEG): container finished" podID="489547ee-81ea-49ee-bc50-63d063aa4e2c" containerID="635a534ed255204f7cf57e80d2d2946e12b9a7dd37d056958bba55cd494bc91a" exitCode=0
Nov 03 21:08:50.358661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.355767    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v" event={"ID":"489547ee-81ea-49ee-bc50-63d063aa4e2c","Type":"ContainerDied","Data":"635a534ed255204f7cf57e80d2d2946e12b9a7dd37d056958bba55cd494bc91a"}
Nov 03 21:08:50.380296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.379972    5533 generic.go:356] "Generic (PLEG): container finished" podID="82a27319-b624-45e4-b778-0e5112e40432" containerID="4f1782955156057a87a68bb1352c68d445175d3d538223e41de40919ad317f91" exitCode=0
Nov 03 21:08:50.380296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.380106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g" event={"ID":"82a27319-b624-45e4-b778-0e5112e40432","Type":"ContainerDied","Data":"4f1782955156057a87a68bb1352c68d445175d3d538223e41de40919ad317f91"}
Nov 03 21:08:50.384271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.383121    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" event={"ID":"110672cc-7b89-4860-a963-d39620d3486a","Type":"ContainerStarted","Data":"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"}
Nov 03 21:08:50.400597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.399739    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6" event={"ID":"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe","Type":"ContainerStarted","Data":"633c823fe9a344d76a8e7f94e89ad1a0161c8521124d524b805a5584e5be9fb0"}
Nov 03 21:08:50.400597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.399810    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6" event={"ID":"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe","Type":"ContainerStarted","Data":"177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a"}
Nov 03 21:08:50.409891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.409851    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" event={"ID":"f6c65baa-882c-4cce-b5c7-5754d1fbff81","Type":"ContainerStarted","Data":"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"}
Nov 03 21:08:50.410043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.409922    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" event={"ID":"f6c65baa-882c-4cce-b5c7-5754d1fbff81","Type":"ContainerStarted","Data":"16a6950fe06151e733d6829493ff52d3ab3ceb393f5e5e16d0bbbadb5c6348f4"}
Nov 03 21:08:50.514404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.514371    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7636c2a5-50a7-484a-b73d-ce006787a3ff" path="/var/lib/kubelet/pods/7636c2a5-50a7-484a-b73d-ce006787a3ff/volumes"
Nov 03 21:08:50.607809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.607700    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podStartSLOduration=2.607681345 podStartE2EDuration="2.607681345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.604974615 +0000 UTC m=+1232.236288105" watchObservedRunningTime="2026-11-03 21:08:50.607681345 +0000 UTC m=+1232.238994822"
Nov 03 21:08:50.849502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.849403    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6" podStartSLOduration=2.849389474 podStartE2EDuration="2.849389474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.845160959 +0000 UTC m=+1232.476474447" watchObservedRunningTime="2026-11-03 21:08:50.849389474 +0000 UTC m=+1232.480702950"
Nov 03 21:08:50.877078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.876988    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:50.877366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.877342    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:50.895876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.877493    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:50.995560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:50.995496    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podStartSLOduration=2.995476657 podStartE2EDuration="2.995476657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:50.9605093 +0000 UTC m=+1232.591822778" watchObservedRunningTime="2026-11-03 21:08:50.995476657 +0000 UTC m=+1232.626790134"
Nov 03 21:08:51.138105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.138029    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:51.139734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.139709    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:51.139850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.139768    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:51.418741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.418662    5533 generic.go:356] "Generic (PLEG): container finished" podID="654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" containerID="633c823fe9a344d76a8e7f94e89ad1a0161c8521124d524b805a5584e5be9fb0" exitCode=0
Nov 03 21:08:51.418884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.418746    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6" event={"ID":"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe","Type":"ContainerDied","Data":"633c823fe9a344d76a8e7f94e89ad1a0161c8521124d524b805a5584e5be9fb0"}
Nov 03 21:08:51.715798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.715775    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:51.742810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.742786    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:51.837780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.837752    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\" (UniqueName: \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\") pod \"489547ee-81ea-49ee-bc50-63d063aa4e2c\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") "
Nov 03 21:08:51.837969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.837800    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\") pod \"489547ee-81ea-49ee-bc50-63d063aa4e2c\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") "
Nov 03 21:08:51.837969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.837850    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\") pod \"489547ee-81ea-49ee-bc50-63d063aa4e2c\" (UID: \"489547ee-81ea-49ee-bc50-63d063aa4e2c\") "
Nov 03 21:08:51.838191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.838167    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca" pod "489547ee-81ea-49ee-bc50-63d063aa4e2c" (UID: "489547ee-81ea-49ee-bc50-63d063aa4e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:51.838546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.838526    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.839994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.839966    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca" pod "489547ee-81ea-49ee-bc50-63d063aa4e2c" (UID: "489547ee-81ea-49ee-bc50-63d063aa4e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:51.840733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.840717    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp" pod "489547ee-81ea-49ee-bc50-63d063aa4e2c" (UID: "489547ee-81ea-49ee-bc50-63d063aa4e2c"). InnerVolumeSpecName "kube-api-access-92sxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:51.889688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.875401    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:51.889688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.875467    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:51.939618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.939515    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\") pod \"82a27319-b624-45e4-b778-0e5112e40432\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") "
Nov 03 21:08:51.939841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.939664    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\") pod \"82a27319-b624-45e4-b778-0e5112e40432\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") "
Nov 03 21:08:51.939841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.939723    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\" (UniqueName: \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\") pod \"82a27319-b624-45e4-b778-0e5112e40432\" (UID: \"82a27319-b624-45e4-b778-0e5112e40432\") "
Nov 03 21:08:51.939841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.939793    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca" pod "82a27319-b624-45e4-b778-0e5112e40432" (UID: "82a27319-b624-45e4-b778-0e5112e40432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:51.940404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.940366    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92sxp\" (UniqueName: \"kubernetes.io/projected/489547ee-81ea-49ee-bc50-63d063aa4e2c-kube-api-access-92sxp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.940404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.940400    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/489547ee-81ea-49ee-bc50-63d063aa4e2c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.940535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.940415    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a27319-b624-45e4-b778-0e5112e40432-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.941743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.941714    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca" pod "82a27319-b624-45e4-b778-0e5112e40432" (UID: "82a27319-b624-45e4-b778-0e5112e40432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:51.942181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:51.942157    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf" pod "82a27319-b624-45e4-b778-0e5112e40432" (UID: "82a27319-b624-45e4-b778-0e5112e40432"). InnerVolumeSpecName "kube-api-access-d5msf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:52.041531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.041501    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a27319-b624-45e4-b778-0e5112e40432-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:52.041531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.041530    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5msf\" (UniqueName: \"kubernetes.io/projected/82a27319-b624-45e4-b778-0e5112e40432-kube-api-access-d5msf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:52.079658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.076425    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"]
Nov 03 21:08:52.104788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.104749    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b0fc0f3-htz69"]
Nov 03 21:08:52.137613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.137587    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:52.137752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.137651    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:52.427789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.427764    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"
Nov 03 21:08:52.428351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.427786    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v" event={"ID":"489547ee-81ea-49ee-bc50-63d063aa4e2c","Type":"ContainerDied","Data":"b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c"}
Nov 03 21:08:52.428351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.427845    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5241822de78261c9da0fb293978d2ea389e2ff33c63d9bd2e10d640d8ffb52c"
Nov 03 21:08:52.429445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.429425    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"
Nov 03 21:08:52.429584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.429420    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g" event={"ID":"82a27319-b624-45e4-b778-0e5112e40432","Type":"ContainerDied","Data":"f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99"}
Nov 03 21:08:52.429584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.429537    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c62e9230e646837688bb88d767f81a18f10caf37dc8f2b2b7773b5ae5e6b99"
Nov 03 21:08:52.527659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.527593    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07b29e4-29cb-48b6-9db7-a0662294c6e6" path="/var/lib/kubelet/pods/d07b29e4-29cb-48b6-9db7-a0662294c6e6/volumes"
Nov 03 21:08:52.745893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.745872    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:52.811277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.811247    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\") pod \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") "
Nov 03 21:08:52.811432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.811296    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\" (UniqueName: \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\") pod \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") "
Nov 03 21:08:52.811432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.811399    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\") pod \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\" (UID: \"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe\") "
Nov 03 21:08:52.811811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.811574    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca" pod "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" (UID: "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:52.811811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.811770    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:52.813520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.813497    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca" pod "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" (UID: "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:52.813594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.813543    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc" pod "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" (UID: "654fd73d-4cf1-41d1-b0a2-ee56728b1cfe"). InnerVolumeSpecName "kube-api-access-vgxwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:52.908673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.899205    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:52.908673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.899271    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:52.912458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.912410    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:52.912458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:52.912436    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxwc\" (UniqueName: \"kubernetes.io/projected/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe-kube-api-access-vgxwc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:53.138248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.138169    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:53.138248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.138225    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:53.437894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.437860    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6" event={"ID":"654fd73d-4cf1-41d1-b0a2-ee56728b1cfe","Type":"ContainerDied","Data":"177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a"}
Nov 03 21:08:53.438274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.437910    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177a0cafc15b72443c3be7c02e663be45e09f0ffd114d305d8ce6c8d965e542a"
Nov 03 21:08:53.438274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.437961    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"
Nov 03 21:08:53.879086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.878996    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:53.879356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:53.879323    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:54.138024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:54.137947    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:54.138024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:54.138005    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:54.883661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:54.882844    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:54.883661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:54.882902    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:55.137998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:55.137923    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:55.137998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:55.137970    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:55.875170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:55.875135    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:08:55.875327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:55.875202    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 21:08:56.138358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:56.138258    5533 patch_prober.go:29] interesting pod/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 21:08:56.138358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:56.138324    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 21:08:56.881431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:56.881395    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:56.881880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:56.881852    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:56.882532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:56.882514    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:08:57.095929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:57.095895    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"]
Nov 03 21:08:57.099289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:57.099262    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fb7af4-gr97v"]
Nov 03 21:08:57.141627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:57.141564    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:57.141987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:57.141838    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:57.142394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:57.142378    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 21:08:58.496977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:58.496944    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be50500-cbda-4852-95ad-79a09fecd980" path="/var/lib/kubelet/pods/7be50500-cbda-4852-95ad-79a09fecd980/volumes"
Nov 03 21:08:59.532570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.532537    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"]
Nov 03 21:08:59.533079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533065    5533 cpu_manager.go:414] "containerMap: removing container" podUID="82a27319-b624-45e4-b778-0e5112e40432" containerName="revoke-keys"
Nov 03 21:08:59.533121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533079    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a27319-b624-45e4-b778-0e5112e40432" containerName="revoke-keys"
Nov 03 21:08:59.533121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533096    5533 cpu_manager.go:414] "containerMap: removing container" podUID="654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" containerName="revoke-keys"
Nov 03 21:08:59.533121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533102    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" containerName="revoke-keys"
Nov 03 21:08:59.533121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533115    5533 cpu_manager.go:414] "containerMap: removing container" podUID="489547ee-81ea-49ee-bc50-63d063aa4e2c" containerName="revoke-keys"
Nov 03 21:08:59.533121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533120    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489547ee-81ea-49ee-bc50-63d063aa4e2c" containerName="revoke-keys"
Nov 03 21:08:59.533272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533192    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a27319-b624-45e4-b778-0e5112e40432" containerName="revoke-keys"
Nov 03 21:08:59.533272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533201    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" containerName="revoke-keys"
Nov 03 21:08:59.533272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.533209    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="489547ee-81ea-49ee-bc50-63d063aa4e2c" containerName="revoke-keys"
Nov 03 21:08:59.537774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.537757    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.586876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.586853    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:59.590541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.590513    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.590670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.590569    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.590768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.590742    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v62\" (UniqueName: \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.633726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.633694    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:08:59.637484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.637461    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"]
Nov 03 21:08:59.691885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.691862    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.692025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.691925    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.692025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.691968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v62\" (UniqueName: \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.692540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.692515    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.694149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.694126    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.749624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.749601    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v62\" (UniqueName: \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\") pod \"maas-api-revoke-sub-4053930a-2fr4x\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:08:59.851217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:08:59.851158    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:09:00.219625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:00.219592    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"]
Nov 03 21:09:00.219873 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:00.219840    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c7df40_1342_4db6_b005_a00a07e2524b.slice/crio-149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc WatchSource:0}: Error finding container 149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc: Status 404 returned error can't find the container with id 149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc
Nov 03 21:09:00.513521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:00.513434    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x" event={"ID":"d7c7df40-1342-4db6-b005-a00a07e2524b","Type":"ContainerStarted","Data":"149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc"}
Nov 03 21:09:01.521142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:01.521110    5533 generic.go:356] "Generic (PLEG): container finished" podID="d7c7df40-1342-4db6-b005-a00a07e2524b" containerID="7619c1480f2bcde3db6bd16e185c205c3dac0abd8d6fedc3f806bbab23e5af55" exitCode=0
Nov 03 21:09:01.521508 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:01.521193    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x" event={"ID":"d7c7df40-1342-4db6-b005-a00a07e2524b","Type":"ContainerDied","Data":"7619c1480f2bcde3db6bd16e185c205c3dac0abd8d6fedc3f806bbab23e5af55"}
Nov 03 21:09:02.669747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.669723    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:09:02.725692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.725670    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\" (UniqueName: \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\") pod \"d7c7df40-1342-4db6-b005-a00a07e2524b\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") "
Nov 03 21:09:02.725844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.725774    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\") pod \"d7c7df40-1342-4db6-b005-a00a07e2524b\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") "
Nov 03 21:09:02.725844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.725816    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\") pod \"d7c7df40-1342-4db6-b005-a00a07e2524b\" (UID: \"d7c7df40-1342-4db6-b005-a00a07e2524b\") "
Nov 03 21:09:02.726079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.726057    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca" pod "d7c7df40-1342-4db6-b005-a00a07e2524b" (UID: "d7c7df40-1342-4db6-b005-a00a07e2524b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:02.726355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.726338    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:02.727769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.727745    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62" pod "d7c7df40-1342-4db6-b005-a00a07e2524b" (UID: "d7c7df40-1342-4db6-b005-a00a07e2524b"). InnerVolumeSpecName "kube-api-access-z8v62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:02.727889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.727827    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca" pod "d7c7df40-1342-4db6-b005-a00a07e2524b" (UID: "d7c7df40-1342-4db6-b005-a00a07e2524b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:02.828034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.827972    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c7df40-1342-4db6-b005-a00a07e2524b-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:02.828034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:02.827997    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v62\" (UniqueName: \"kubernetes.io/projected/d7c7df40-1342-4db6-b005-a00a07e2524b-kube-api-access-z8v62\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:03.536294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:03.536265    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"
Nov 03 21:09:03.536460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:03.536300    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x" event={"ID":"d7c7df40-1342-4db6-b005-a00a07e2524b","Type":"ContainerDied","Data":"149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc"}
Nov 03 21:09:03.536460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:03.536347    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149ccb6de18b731df402a5169ffac64d3f8e14295fcdb138846268a84ab542bc"
Nov 03 21:09:04.548593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.548561    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"]
Nov 03 21:09:04.549284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.549262    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d7c7df40-1342-4db6-b005-a00a07e2524b" containerName="revoke-keys"
Nov 03 21:09:04.549379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.549285    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c7df40-1342-4db6-b005-a00a07e2524b" containerName="revoke-keys"
Nov 03 21:09:04.549450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.549427    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c7df40-1342-4db6-b005-a00a07e2524b" containerName="revoke-keys"
Nov 03 21:09:04.553277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.553253    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.602414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.602388    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:04.649486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.649456    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:09:04.653304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.653281    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"]
Nov 03 21:09:04.659100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.659074    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tprr\" (UniqueName: \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.659269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.659251    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.659316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.659296    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.760626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.760594    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.760772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.760660    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.760829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.760804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tprr\" (UniqueName: \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.761306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.761283    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.763154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.763123    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.818169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.818113    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tprr\" (UniqueName: \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\") pod \"maas-api-revoke-sub-f03af4e7-ktk4n\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:04.870732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:04.870708    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:05.046975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:05.046936    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"]
Nov 03 21:09:05.056049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:05.056010    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcb5a1-mqtdh"]
Nov 03 21:09:05.241796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:05.241769    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"]
Nov 03 21:09:05.242276 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:05.242253    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b16a8c1_7b0d_4b99_ae73_b3779c91ec1f.slice/crio-e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc WatchSource:0}: Error finding container e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc: Status 404 returned error can't find the container with id e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc
Nov 03 21:09:05.552659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:05.552596    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n" event={"ID":"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f","Type":"ContainerStarted","Data":"e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc"}
Nov 03 21:09:06.497549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:06.497505    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a168e771-2b8c-4c44-8cd7-f040996a98d1" path="/var/lib/kubelet/pods/a168e771-2b8c-4c44-8cd7-f040996a98d1/volumes"
Nov 03 21:09:06.562140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:06.562110    5533 generic.go:356] "Generic (PLEG): container finished" podID="5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" containerID="d8ac4cb42f3ce6cd0f37f5ede1f1bc5b0a2061a8a1cb15f0ffad50bd2788f4ce" exitCode=0
Nov 03 21:09:06.562550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:06.562186    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n" event={"ID":"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f","Type":"ContainerDied","Data":"d8ac4cb42f3ce6cd0f37f5ede1f1bc5b0a2061a8a1cb15f0ffad50bd2788f4ce"}
Nov 03 21:09:07.046416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.046372    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"]
Nov 03 21:09:07.050726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.050700    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0b73fb-gvfjz"]
Nov 03 21:09:07.182433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.182400    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:09:07.765097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.765072    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:07.915289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.914164    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\" (UniqueName: \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\") pod \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") "
Nov 03 21:09:07.915289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.914268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\") pod \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") "
Nov 03 21:09:07.915289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.914399    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\") pod \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\" (UID: \"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f\") "
Nov 03 21:09:07.954267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.928682    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca" pod "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" (UID: "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:07.954267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:07.941563    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr" pod "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" (UID: "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f"). InnerVolumeSpecName "kube-api-access-6tprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:08.058659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.058198    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tprr\" (UniqueName: \"kubernetes.io/projected/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-kube-api-access-6tprr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.058659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.058236    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.058659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.058351    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca" pod "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" (UID: "5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:08.178709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.161916    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:08.519740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.517731    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12989a6-aad4-4c02-95be-887268f5d242" path="/var/lib/kubelet/pods/c12989a6-aad4-4c02-95be-887268f5d242/volumes"
Nov 03 21:09:08.604304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.604266    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n" event={"ID":"5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f","Type":"ContainerDied","Data":"e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc"}
Nov 03 21:09:08.604560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.604541    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c696f2639d3691eab06ba7258d771719f5ce47422685219633fa8b442c9cdc"
Nov 03 21:09:08.604784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:08.604769    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"
Nov 03 21:09:09.646658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:09.645735    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:09.841764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:09.841719    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:10.016764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.016719    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:10.119671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.119612    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:10.480988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.480935    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podUID="3b4da839-0fd8-4d9e-81a2-371496377bba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 21:09:10.685687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.646437    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 21:09:10.963666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.843235    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:10.990474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.989777    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:09:11.006758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:10.999112    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:09:11.127663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.030953    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 21:09:11.127663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.121546    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:09:11.127663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.126800    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:11.127663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.127158    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:09:11.257114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.257022    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 21:09:11.268157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.268125    5533 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:11.489625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.489577    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq" podUID="3b4da839-0fd8-4d9e-81a2-371496377bba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:11.490290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.490268    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 21:09:11.539198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.539117    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 21:09:11.632318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.614606    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 21:09:11.633781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.633751    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 21:09:11.648599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.648574    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq"
Nov 03 21:09:11.681508 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:11.652404    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 21:09:15.283576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.283540    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"]
Nov 03 21:09:15.284120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.284106    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" containerName="revoke-keys"
Nov 03 21:09:15.284166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.284121    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" containerName="revoke-keys"
Nov 03 21:09:15.284207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.284202    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" containerName="revoke-keys"
Nov 03 21:09:15.289120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.289102    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.342760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.342726    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"]
Nov 03 21:09:15.353363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.353321    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:15.384249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.384225    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:09:15.459954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.459922    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kbh\" (UniqueName: \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.460099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.459993    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.460191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.460166    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.561350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.561271    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kbh\" (UniqueName: \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.561506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.561385    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.561506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.561492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.562078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.562055    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.563847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.563824    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.618445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.618421    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kbh\" (UniqueName: \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\") pod \"maas-api-revoke-sub-bed7d42a-s6wvn\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:15.903380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:15.903320    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:16.280008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:16.279978    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"]
Nov 03 21:09:16.280791 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:16.280761    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12b0762_623c_4f19_bed8_e1f640db0f11.slice/crio-89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd WatchSource:0}: Error finding container 89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd: Status 404 returned error can't find the container with id 89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd
Nov 03 21:09:16.995048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:16.994836    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn" event={"ID":"f12b0762-623c-4f19-bed8-e1f640db0f11","Type":"ContainerStarted","Data":"a557d0170b467b8efdc35f8083b4a5a09a64627805fbafcfa9daafe4e56edad4"}
Nov 03 21:09:16.995048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:16.994903    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn" event={"ID":"f12b0762-623c-4f19-bed8-e1f640db0f11","Type":"ContainerStarted","Data":"89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd"}
Nov 03 21:09:17.104986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:17.104936    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn" podStartSLOduration=2.104923077 podStartE2EDuration="2.104923077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:17.103078416 +0000 UTC m=+1258.734391905" watchObservedRunningTime="2026-11-03 21:09:17.104923077 +0000 UTC m=+1258.736236555"
Nov 03 21:09:18.006226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:18.006189    5533 generic.go:356] "Generic (PLEG): container finished" podID="f12b0762-623c-4f19-bed8-e1f640db0f11" containerID="a557d0170b467b8efdc35f8083b4a5a09a64627805fbafcfa9daafe4e56edad4" exitCode=0
Nov 03 21:09:18.006614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:18.006266    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn" event={"ID":"f12b0762-623c-4f19-bed8-e1f640db0f11","Type":"ContainerDied","Data":"a557d0170b467b8efdc35f8083b4a5a09a64627805fbafcfa9daafe4e56edad4"}
Nov 03 21:09:18.768398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:18.768368    5533 scope.go:122] "RemoveContainer" containerID="601ceb1a73959e05b135a3c9246ea525227d9cb439c028aeb3e4f5cc5f4a25e7"
Nov 03 21:09:19.171492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.171460    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:19.206513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.206487    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\") pod \"f12b0762-623c-4f19-bed8-e1f640db0f11\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") "
Nov 03 21:09:19.206646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.206625    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\") pod \"f12b0762-623c-4f19-bed8-e1f640db0f11\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") "
Nov 03 21:09:19.206699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.206667    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\" (UniqueName: \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\") pod \"f12b0762-623c-4f19-bed8-e1f640db0f11\" (UID: \"f12b0762-623c-4f19-bed8-e1f640db0f11\") "
Nov 03 21:09:19.206866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.206841    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca" pod "f12b0762-623c-4f19-bed8-e1f640db0f11" (UID: "f12b0762-623c-4f19-bed8-e1f640db0f11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:19.207087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.207064    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:19.208797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.208771    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca" pod "f12b0762-623c-4f19-bed8-e1f640db0f11" (UID: "f12b0762-623c-4f19-bed8-e1f640db0f11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:19.208879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.208812    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh" pod "f12b0762-623c-4f19-bed8-e1f640db0f11" (UID: "f12b0762-623c-4f19-bed8-e1f640db0f11"). InnerVolumeSpecName "kube-api-access-79kbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:19.260736 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:09:19.260701    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:09:19.309054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.309019    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12b0762-623c-4f19-bed8-e1f640db0f11-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:19.309195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:19.309059    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kbh\" (UniqueName: \"kubernetes.io/projected/f12b0762-623c-4f19-bed8-e1f640db0f11-kube-api-access-79kbh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:20.022850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:20.022817    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"
Nov 03 21:09:20.023071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:20.022819    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn" event={"ID":"f12b0762-623c-4f19-bed8-e1f640db0f11","Type":"ContainerDied","Data":"89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd"}
Nov 03 21:09:20.023071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:20.022967    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89799d557477a1e5212106d3199a2212c9dafc950781cf81ee92eb5070e285bd"
Nov 03 21:09:22.646772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.646732    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:09:22.647698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.647678    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f12b0762-623c-4f19-bed8-e1f640db0f11" containerName="revoke-keys"
Nov 03 21:09:22.647698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.647697    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12b0762-623c-4f19-bed8-e1f640db0f11" containerName="revoke-keys"
Nov 03 21:09:22.647897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.647794    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12b0762-623c-4f19-bed8-e1f640db0f11" containerName="revoke-keys"
Nov 03 21:09:22.652445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.652426    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.706855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.706823    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:09:22.747097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.747060    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-32f6e43c\""
Nov 03 21:09:22.747282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.747140    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-32f6e43c-dockercfg-q28zk\""
Nov 03 21:09:22.763206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.763174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjw8\" (UniqueName: \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.763357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.763308    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.864658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.864515    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.864848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.864791    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjw8\" (UniqueName: \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.873336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.873293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.930401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.930360    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjw8\" (UniqueName: \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\") pod \"payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:22.969363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:22.969333    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:23.345518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:23.345486    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:09:23.346068 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:23.346016    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caf9cd6_f035_4717_bb49_81b25587c968.slice/crio-a10aac484ce76c7edf0a74921cfbeab15f0f814286ae2b8f2d49bac0c4d6165d WatchSource:0}: Error finding container a10aac484ce76c7edf0a74921cfbeab15f0f814286ae2b8f2d49bac0c4d6165d: Status 404 returned error can't find the container with id a10aac484ce76c7edf0a74921cfbeab15f0f814286ae2b8f2d49bac0c4d6165d
Nov 03 21:09:24.059984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:24.059948    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" event={"ID":"3caf9cd6-f035-4717-bb49-81b25587c968","Type":"ContainerStarted","Data":"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"}
Nov 03 21:09:24.060386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:24.060000    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" event={"ID":"3caf9cd6-f035-4717-bb49-81b25587c968","Type":"ContainerStarted","Data":"a10aac484ce76c7edf0a74921cfbeab15f0f814286ae2b8f2d49bac0c4d6165d"}
Nov 03 21:09:24.060386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:24.060087    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:24.171718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:24.171659    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" podStartSLOduration=2.171627843 podStartE2EDuration="2.171627843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:24.170034567 +0000 UTC m=+1265.801348090" watchObservedRunningTime="2026-11-03 21:09:24.171627843 +0000 UTC m=+1265.802941323"
Nov 03 21:09:30.550714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:30.549706    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 21:09:30.639123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:30.638715    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:31.178655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:31.177709    5533 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:09:34.827890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.827856    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"]
Nov 03 21:09:34.832256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.832235    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.851148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.851125    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.851251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.851235    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.851353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.851341    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlctr\" (UniqueName: \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.882016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.881983    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:34.887171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.887145    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"]
Nov 03 21:09:34.928825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.928803    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:09:34.952763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.952738    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlctr\" (UniqueName: \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.952868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.952817    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.952930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.952895    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.953512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.953494    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:34.955116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:34.955098    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:35.010039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:35.010019    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlctr\" (UniqueName: \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\") pod \"maas-api-revoke-sub-51b68627-8rfkf\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:35.082433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:35.082372    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:09:35.146127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:35.146104    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:35.536593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:35.536563    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"]
Nov 03 21:09:35.537597 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:35.537558    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a4f880_b67e_4aef_9b16_07c3c79e1f59.slice/crio-e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c WatchSource:0}: Error finding container e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c: Status 404 returned error can't find the container with id e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c
Nov 03 21:09:36.353944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:36.353910    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf" event={"ID":"c8a4f880-b67e-4aef-9b16-07c3c79e1f59","Type":"ContainerStarted","Data":"54da0ba7af28ed68a32a1237e1b77b05cf296325846f0a48ce4ab77a2838a002"}
Nov 03 21:09:36.354338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:36.353966    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf" event={"ID":"c8a4f880-b67e-4aef-9b16-07c3c79e1f59","Type":"ContainerStarted","Data":"e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c"}
Nov 03 21:09:36.462761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:36.462718    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf" podStartSLOduration=2.462704168 podStartE2EDuration="2.462704168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:36.462056385 +0000 UTC m=+1278.093369862" watchObservedRunningTime="2026-11-03 21:09:36.462704168 +0000 UTC m=+1278.094017644"
Nov 03 21:09:37.364088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:37.364049    5533 generic.go:356] "Generic (PLEG): container finished" podID="c8a4f880-b67e-4aef-9b16-07c3c79e1f59" containerID="54da0ba7af28ed68a32a1237e1b77b05cf296325846f0a48ce4ab77a2838a002" exitCode=0
Nov 03 21:09:37.364484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:37.364131    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf" event={"ID":"c8a4f880-b67e-4aef-9b16-07c3c79e1f59","Type":"ContainerDied","Data":"54da0ba7af28ed68a32a1237e1b77b05cf296325846f0a48ce4ab77a2838a002"}
Nov 03 21:09:38.529123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.529101    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:38.595850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.595820    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\" (UniqueName: \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\") pod \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") "
Nov 03 21:09:38.596061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.596042    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\") pod \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") "
Nov 03 21:09:38.596120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.596086    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\") pod \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\" (UID: \"c8a4f880-b67e-4aef-9b16-07c3c79e1f59\") "
Nov 03 21:09:38.596479 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.596456    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca" pod "c8a4f880-b67e-4aef-9b16-07c3c79e1f59" (UID: "c8a4f880-b67e-4aef-9b16-07c3c79e1f59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:38.596721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.596703    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:38.598011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.597987    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca" pod "c8a4f880-b67e-4aef-9b16-07c3c79e1f59" (UID: "c8a4f880-b67e-4aef-9b16-07c3c79e1f59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:38.598304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.598290    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr" pod "c8a4f880-b67e-4aef-9b16-07c3c79e1f59" (UID: "c8a4f880-b67e-4aef-9b16-07c3c79e1f59"). InnerVolumeSpecName "kube-api-access-rlctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:38.698532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.698501    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlctr\" (UniqueName: \"kubernetes.io/projected/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-kube-api-access-rlctr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:38.698724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:38.698535    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a4f880-b67e-4aef-9b16-07c3c79e1f59-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:39.383471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:39.383437    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"
Nov 03 21:09:39.383675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:39.383474    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf" event={"ID":"c8a4f880-b67e-4aef-9b16-07c3c79e1f59","Type":"ContainerDied","Data":"e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c"}
Nov 03 21:09:39.383675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:39.383522    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e29cbab241becb5ae36511d2220c196608700724aae3d1ade22b8f99338a9c"
Nov 03 21:09:41.518157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:41.518121    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:09:42.578627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.578586    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"]
Nov 03 21:09:42.579331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.579309    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c8a4f880-b67e-4aef-9b16-07c3c79e1f59" containerName="revoke-keys"
Nov 03 21:09:42.579429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.579330    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a4f880-b67e-4aef-9b16-07c3c79e1f59" containerName="revoke-keys"
Nov 03 21:09:42.579531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.579517    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a4f880-b67e-4aef-9b16-07c3c79e1f59" containerName="revoke-keys"
Nov 03 21:09:42.583152 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.583128    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.652319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.652281    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:42.678865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.661451    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.679205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.679173    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.686486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.679553    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf9r\" (UniqueName: \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.740818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.740769    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:09:42.789525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.789485    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.789713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.789571    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.789713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.789691    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf9r\" (UniqueName: \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.790888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.790737    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"]
Nov 03 21:09:42.806146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.806116    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.824563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.824514    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.853516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.853410    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf9r\" (UniqueName: \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\") pod \"maas-api-revoke-sub-3d5cfe1c-z64c9\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:42.920163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:42.920123    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:43.324421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:43.323387    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"]
Nov 03 21:09:43.426182 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:43.426148    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9" event={"ID":"8513bd7f-3494-4cac-a2f2-c15c38d86f57","Type":"ContainerStarted","Data":"99f54566756fe8748cc78b22db2d8532b20413a3579308ddf6264e89ad0ab8c9"}
Nov 03 21:09:44.437844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:44.437795    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9" event={"ID":"8513bd7f-3494-4cac-a2f2-c15c38d86f57","Type":"ContainerStarted","Data":"a8dfb154fcf1a52f42edd5206b156c93b840101f502d7a7deabf3fe48874be8c"}
Nov 03 21:09:44.583659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:44.582667    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9" podStartSLOduration=2.582647921 podStartE2EDuration="2.582647921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:44.562979812 +0000 UTC m=+1286.194293290" watchObservedRunningTime="2026-11-03 21:09:44.582647921 +0000 UTC m=+1286.213961390"
Nov 03 21:09:45.049468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:45.049386    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"]
Nov 03 21:09:45.067659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:45.066568    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6611650-5t6cv"]
Nov 03 21:09:45.447013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:45.446983    5533 generic.go:356] "Generic (PLEG): container finished" podID="8513bd7f-3494-4cac-a2f2-c15c38d86f57" containerID="a8dfb154fcf1a52f42edd5206b156c93b840101f502d7a7deabf3fe48874be8c" exitCode=0
Nov 03 21:09:45.447429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:45.447097    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9" event={"ID":"8513bd7f-3494-4cac-a2f2-c15c38d86f57","Type":"ContainerDied","Data":"a8dfb154fcf1a52f42edd5206b156c93b840101f502d7a7deabf3fe48874be8c"}
Nov 03 21:09:46.438436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.438406    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"]
Nov 03 21:09:46.441954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.441938    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.476760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.476728    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.477175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.476885    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.477175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.476930    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6jl\" (UniqueName: \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.496819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.496788    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1eb4b1-0efd-4491-bfde-3016b95abd8c" path="/var/lib/kubelet/pods/eb1eb4b1-0efd-4491-bfde-3016b95abd8c/volumes"
Nov 03 21:09:46.498778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.498749    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"]
Nov 03 21:09:46.595660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.581393    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.595660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.581530    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.595660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.581578    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6jl\" (UniqueName: \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.595660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.582909    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.618719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.618688    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.670819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.659363    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6jl\" (UniqueName: \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\") pod \"maas-api-revoke-sub-766d02c0-fbpsz\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.742252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.742225    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:46.756597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.756569    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:46.789795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.789753    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\") pod \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") "
Nov 03 21:09:46.789936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.789918    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\") pod \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") "
Nov 03 21:09:46.790014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.789989    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\" (UniqueName: \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\") pod \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\" (UID: \"8513bd7f-3494-4cac-a2f2-c15c38d86f57\") "
Nov 03 21:09:46.790312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.790290    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca" pod "8513bd7f-3494-4cac-a2f2-c15c38d86f57" (UID: "8513bd7f-3494-4cac-a2f2-c15c38d86f57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:46.790531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.790514    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:46.792319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.792291    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca" pod "8513bd7f-3494-4cac-a2f2-c15c38d86f57" (UID: "8513bd7f-3494-4cac-a2f2-c15c38d86f57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:46.792612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.792592    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r" pod "8513bd7f-3494-4cac-a2f2-c15c38d86f57" (UID: "8513bd7f-3494-4cac-a2f2-c15c38d86f57"). InnerVolumeSpecName "kube-api-access-sqf9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:46.892057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.892022    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf9r\" (UniqueName: \"kubernetes.io/projected/8513bd7f-3494-4cac-a2f2-c15c38d86f57-kube-api-access-sqf9r\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:46.892057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:46.892054    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8513bd7f-3494-4cac-a2f2-c15c38d86f57-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:47.150798 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:47.150764    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode381d43c_8f06_494e_813e_44dfd11b988a.slice/crio-620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea WatchSource:0}: Error finding container 620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea: Status 404 returned error can't find the container with id 620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea
Nov 03 21:09:47.150933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:47.150862    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"]
Nov 03 21:09:47.464382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:47.464351    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"
Nov 03 21:09:47.464479 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:47.464359    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9" event={"ID":"8513bd7f-3494-4cac-a2f2-c15c38d86f57","Type":"ContainerDied","Data":"99f54566756fe8748cc78b22db2d8532b20413a3579308ddf6264e89ad0ab8c9"}
Nov 03 21:09:47.464541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:47.464526    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f54566756fe8748cc78b22db2d8532b20413a3579308ddf6264e89ad0ab8c9"
Nov 03 21:09:47.466576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:47.466547    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz" event={"ID":"e381d43c-8f06-494e-813e-44dfd11b988a","Type":"ContainerStarted","Data":"620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea"}
Nov 03 21:09:48.477063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:48.477031    5533 generic.go:356] "Generic (PLEG): container finished" podID="e381d43c-8f06-494e-813e-44dfd11b988a" containerID="e4a5c757a72b8446cb99949825a82fb618477bef61cac74a9a82c31d6b3529cb" exitCode=0
Nov 03 21:09:48.477529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:48.477113    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz" event={"ID":"e381d43c-8f06-494e-813e-44dfd11b988a","Type":"ContainerDied","Data":"e4a5c757a72b8446cb99949825a82fb618477bef61cac74a9a82c31d6b3529cb"}
Nov 03 21:09:49.260166 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:09:49.260132    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:09:49.626898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.626876    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:49.724415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.724357    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\") pod \"e381d43c-8f06-494e-813e-44dfd11b988a\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") "
Nov 03 21:09:49.724578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.724435    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\" (UniqueName: \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\") pod \"e381d43c-8f06-494e-813e-44dfd11b988a\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") "
Nov 03 21:09:49.724578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.724534    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\") pod \"e381d43c-8f06-494e-813e-44dfd11b988a\" (UID: \"e381d43c-8f06-494e-813e-44dfd11b988a\") "
Nov 03 21:09:49.724749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.724728    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca" pod "e381d43c-8f06-494e-813e-44dfd11b988a" (UID: "e381d43c-8f06-494e-813e-44dfd11b988a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:49.725054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.725034    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:49.726522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.726496    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca" pod "e381d43c-8f06-494e-813e-44dfd11b988a" (UID: "e381d43c-8f06-494e-813e-44dfd11b988a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:49.726522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.726517    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl" pod "e381d43c-8f06-494e-813e-44dfd11b988a" (UID: "e381d43c-8f06-494e-813e-44dfd11b988a"). InnerVolumeSpecName "kube-api-access-7c6jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:49.826608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.826531    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6jl\" (UniqueName: \"kubernetes.io/projected/e381d43c-8f06-494e-813e-44dfd11b988a-kube-api-access-7c6jl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:49.826608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:49.826559    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e381d43c-8f06-494e-813e-44dfd11b988a-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:50.493837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:50.493810    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"
Nov 03 21:09:50.494068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:50.494027    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz" event={"ID":"e381d43c-8f06-494e-813e-44dfd11b988a","Type":"ContainerDied","Data":"620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea"}
Nov 03 21:09:50.494141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:50.494077    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="620b6b980748ed67047f399e8179016474dc8ebe35734c169cf2675c8925b1ea"
Nov 03 21:09:56.886660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.886612    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"]
Nov 03 21:09:56.887193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887178    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8513bd7f-3494-4cac-a2f2-c15c38d86f57" containerName="revoke-keys"
Nov 03 21:09:56.887193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887194    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8513bd7f-3494-4cac-a2f2-c15c38d86f57" containerName="revoke-keys"
Nov 03 21:09:56.887316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887227    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e381d43c-8f06-494e-813e-44dfd11b988a" containerName="revoke-keys"
Nov 03 21:09:56.887316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887234    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e381d43c-8f06-494e-813e-44dfd11b988a" containerName="revoke-keys"
Nov 03 21:09:56.887316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887312    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8513bd7f-3494-4cac-a2f2-c15c38d86f57" containerName="revoke-keys"
Nov 03 21:09:56.887406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.887323    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e381d43c-8f06-494e-813e-44dfd11b988a" containerName="revoke-keys"
Nov 03 21:09:56.891113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.891091    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:56.944454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.944429    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"]
Nov 03 21:09:56.973898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.973874    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:56.986185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:56.986164    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:09:57.010699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.010671    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqv7\" (UniqueName: \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.010883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.010706    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.010883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.010734    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.111718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.111679    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqv7\" (UniqueName: \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.111718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.111721    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.111961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.111747    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.112513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.112482    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.116804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.116778    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.170254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.170229    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqv7\" (UniqueName: \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\") pod \"maas-api-revoke-sub-d5c957e5-jjzh2\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.205787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.205754    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:09:57.580853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:57.580818    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"]
Nov 03 21:09:57.582213 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:09:57.582182    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eea2f5e_18f9_4814_ae68_1293dd23070d.slice/crio-4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68 WatchSource:0}: Error finding container 4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68: Status 404 returned error can't find the container with id 4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68
Nov 03 21:09:58.714658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:58.699174    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2" event={"ID":"8eea2f5e-18f9-4814-ae68-1293dd23070d","Type":"ContainerStarted","Data":"934d2a91b7911b4d4eb4d43c143ab7b509ea87273537123cfe59ae0e6200836f"}
Nov 03 21:09:58.714658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:58.699261    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2" event={"ID":"8eea2f5e-18f9-4814-ae68-1293dd23070d","Type":"ContainerStarted","Data":"4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68"}
Nov 03 21:09:59.210842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.210809    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:09:59.216946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.216908    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.258776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.258736    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.259213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.259178    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8wr\" (UniqueName: \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.366526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.366488    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-145cc5-dockercfg-ncfdw\""
Nov 03 21:09:59.366895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.366871    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-145cc5\""
Nov 03 21:09:59.373677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.369126    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.417655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.393719    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.417655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.393978    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8wr\" (UniqueName: \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.693658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.689737    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8wr\" (UniqueName: \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\") pod \"payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") " pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.867382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.867093    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:09:59.869228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:09:59.869202    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:10:00.098039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.098004    5533 generic.go:356] "Generic (PLEG): container finished" podID="8eea2f5e-18f9-4814-ae68-1293dd23070d" containerID="934d2a91b7911b4d4eb4d43c143ab7b509ea87273537123cfe59ae0e6200836f" exitCode=0
Nov 03 21:10:00.106857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.106806    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2" event={"ID":"8eea2f5e-18f9-4814-ae68-1293dd23070d","Type":"ContainerDied","Data":"934d2a91b7911b4d4eb4d43c143ab7b509ea87273537123cfe59ae0e6200836f"}
Nov 03 21:10:00.107123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.107087    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:10:00.249401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.249372    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:10:00.249740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.249699    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:10:00.252792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.252762    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.258417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.258395    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:10:00.258669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.258651    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.352681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.349777    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6dacc6-openshift-default-dockercfg-6k8p2\""
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382448    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382565    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382592    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382625    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5kg\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382684    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltc6\" (UniqueName: \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382726    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382773    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.382839    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.386315    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.386517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.386389    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502200    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502259    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502296    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5kg\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltc6\" (UniqueName: \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502377    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502422    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502482    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502527    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.505688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.502587    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.517605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.516957    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.523946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.523303    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.523946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.523481    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.524144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.524101    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.524320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.524285    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.524396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.524319    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.524396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.524331    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.525108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.525073    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.529672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.527124    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.529672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.529168    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.574070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.574036    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltc6\" (UniqueName: \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\") pod \"payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.583704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.582112    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.583704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.583300    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5kg\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\") pod \"e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") " pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.654969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.654657    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:10:00.700484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.700431    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:00.718073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.718044    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:00.726280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.726252    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\") pod \"8eea2f5e-18f9-4814-ae68-1293dd23070d\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") "
Nov 03 21:10:00.726393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.726356    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\" (UniqueName: \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\") pod \"8eea2f5e-18f9-4814-ae68-1293dd23070d\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") "
Nov 03 21:10:00.726542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.726527    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\") pod \"8eea2f5e-18f9-4814-ae68-1293dd23070d\" (UID: \"8eea2f5e-18f9-4814-ae68-1293dd23070d\") "
Nov 03 21:10:00.726680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.726647    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca" pod "8eea2f5e-18f9-4814-ae68-1293dd23070d" (UID: "8eea2f5e-18f9-4814-ae68-1293dd23070d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:00.726928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.726901    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.728989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.728963    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7" pod "8eea2f5e-18f9-4814-ae68-1293dd23070d" (UID: "8eea2f5e-18f9-4814-ae68-1293dd23070d"). InnerVolumeSpecName "kube-api-access-jrqv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:00.729085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.729052    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca" pod "8eea2f5e-18f9-4814-ae68-1293dd23070d" (UID: "8eea2f5e-18f9-4814-ae68-1293dd23070d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:00.759460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.759380    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:10:00.786600 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:10:00.785797    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b668561_2741_4627_a775_08de62fbf073.slice/crio-49feada38265c21d02aa18185b758870d6560bb4b7bec2f9b7927101935b8dc2 WatchSource:0}: Error finding container 49feada38265c21d02aa18185b758870d6560bb4b7bec2f9b7927101935b8dc2: Status 404 returned error can't find the container with id 49feada38265c21d02aa18185b758870d6560bb4b7bec2f9b7927101935b8dc2
Nov 03 21:10:00.827736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.827712    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eea2f5e-18f9-4814-ae68-1293dd23070d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.827736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:00.827735    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqv7\" (UniqueName: \"kubernetes.io/projected/8eea2f5e-18f9-4814-ae68-1293dd23070d-kube-api-access-jrqv7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:01.106269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.106236    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:10:01.120466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.120435    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" event={"ID":"4b668561-2741-4627-a775-08de62fbf073","Type":"ContainerStarted","Data":"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"}
Nov 03 21:10:01.120610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.120503    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:10:01.120610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.120529    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" event={"ID":"4b668561-2741-4627-a775-08de62fbf073","Type":"ContainerStarted","Data":"49feada38265c21d02aa18185b758870d6560bb4b7bec2f9b7927101935b8dc2"}
Nov 03 21:10:01.121806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.121782    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" event={"ID":"78befc1d-adf6-499b-a498-aee6791cf275","Type":"ContainerStarted","Data":"bd4b2d3e0e2cd42927e97d95bb1e2f9a5d202cb9f64fbebca45fabbb0dbce1ba"}
Nov 03 21:10:01.123434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.123409    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2" event={"ID":"8eea2f5e-18f9-4814-ae68-1293dd23070d","Type":"ContainerDied","Data":"4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68"}
Nov 03 21:10:01.123522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.123447    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7773f65e6724aa7cc3f518f73630ac2b449f4a2fff90380c4e2456b4a6ae68"
Nov 03 21:10:01.123522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.123451    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"
Nov 03 21:10:01.152944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.152922    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:10:01.153355 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:10:01.153328    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec14b1d_ebe4_4d84_bc2c_f2b6822d33b6.slice/crio-8eafc05f6c05065199b976d969d218de9d7397525c75923852f5a3bf49c5327b WatchSource:0}: Error finding container 8eafc05f6c05065199b976d969d218de9d7397525c75923852f5a3bf49c5327b: Status 404 returned error can't find the container with id 8eafc05f6c05065199b976d969d218de9d7397525c75923852f5a3bf49c5327b
Nov 03 21:10:01.156099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.156071    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:10:01.156185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.156143    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:10:01.156252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.156187    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:10:01.232549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:01.232491    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" podStartSLOduration=2.232472624 podStartE2EDuration="2.232472624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:01.229157537 +0000 UTC m=+1302.860471025" watchObservedRunningTime="2026-11-03 21:10:01.232472624 +0000 UTC m=+1302.863786102"
Nov 03 21:10:02.132438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.132398    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerStarted","Data":"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"}
Nov 03 21:10:02.132854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.132456    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerStarted","Data":"8eafc05f6c05065199b976d969d218de9d7397525c75923852f5a3bf49c5327b"}
Nov 03 21:10:02.134131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.134102    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" event={"ID":"78befc1d-adf6-499b-a498-aee6791cf275","Type":"ContainerStarted","Data":"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"}
Nov 03 21:10:02.134284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.134169    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:02.251054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.250993    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podStartSLOduration=3.25097564 podStartE2EDuration="3.25097564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:02.246731164 +0000 UTC m=+1303.878044638" watchObservedRunningTime="2026-11-03 21:10:02.25097564 +0000 UTC m=+1303.882289117"
Nov 03 21:10:02.358087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.358035    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" podStartSLOduration=3.358021859 podStartE2EDuration="3.358021859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:02.355127308 +0000 UTC m=+1303.986440784" watchObservedRunningTime="2026-11-03 21:10:02.358021859 +0000 UTC m=+1303.989335352"
Nov 03 21:10:02.718817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.718778    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:02.724123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:02.724100    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:03.141989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:03.141897    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:03.142887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:03.142866    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:10:07.172448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.169130    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:07.172448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.170031    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8eea2f5e-18f9-4814-ae68-1293dd23070d" containerName="revoke-keys"
Nov 03 21:10:07.172448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.170051    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eea2f5e-18f9-4814-ae68-1293dd23070d" containerName="revoke-keys"
Nov 03 21:10:07.172448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.170176    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eea2f5e-18f9-4814-ae68-1293dd23070d" containerName="revoke-keys"
Nov 03 21:10:07.174272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.174243    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.225689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.225657    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:07.226590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.226563    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9h78\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.226733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.226701    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.226812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.226797    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.226874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.226863    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.327929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.327881    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.328110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.328038    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.328192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.328142    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.328192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.328170    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9h78\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.328736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.328705    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.329371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.329339    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.331878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.331854    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.388435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.388394    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9h78\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\") pod \"payload-processing-external-model-7cd67c9464-4lhmm\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") " pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.491616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.491540    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:07.863532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:07.863505    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:07.863532 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:10:07.863512    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d288560_27b9_45a8_ba81_5cd9d1af6597.slice/crio-ad0b08791ed40f2fd7827a013dd6d3623e161432be23e39f16d04e12822d0b6d WatchSource:0}: Error finding container ad0b08791ed40f2fd7827a013dd6d3623e161432be23e39f16d04e12822d0b6d: Status 404 returned error can't find the container with id ad0b08791ed40f2fd7827a013dd6d3623e161432be23e39f16d04e12822d0b6d
Nov 03 21:10:08.186308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:08.186278    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" event={"ID":"8d288560-27b9-45a8-ba81-5cd9d1af6597","Type":"ContainerStarted","Data":"cd31aa70931df46fd1792172017b982b3c2ef1acc1ab3e992d61626d179b8102"}
Nov 03 21:10:08.186664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:08.186335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" event={"ID":"8d288560-27b9-45a8-ba81-5cd9d1af6597","Type":"ContainerStarted","Data":"ad0b08791ed40f2fd7827a013dd6d3623e161432be23e39f16d04e12822d0b6d"}
Nov 03 21:10:08.186664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:08.186449    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:08.297239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:08.297184    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" podStartSLOduration=1.297169232 podStartE2EDuration="1.297169232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:08.294836436 +0000 UTC m=+1309.926149913" watchObservedRunningTime="2026-11-03 21:10:08.297169232 +0000 UTC m=+1309.928482735"
Nov 03 21:10:12.137328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:12.137302    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:10:13.145463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:13.145432    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:10:14.003324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:14.003289    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:14.006176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:14.006145    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:14.049969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:14.049942    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:10:18.798488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.798454    5533 scope.go:122] "RemoveContainer" containerID="d76c73a735b430bc0bd167a8eb8456f58e3addf4b4e2db2054d0586766aa7f78"
Nov 03 21:10:18.808708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.808686    5533 scope.go:122] "RemoveContainer" containerID="b4a35b26abe0e8756ba90fcc276c400298b629a7763831393387423cb7042c76"
Nov 03 21:10:18.817335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.817317    5533 scope.go:122] "RemoveContainer" containerID="473eba31932c35733452b253e3685420e3fab87240af5d4d8c01cb5d157266d0"
Nov 03 21:10:18.826116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.826101    5533 scope.go:122] "RemoveContainer" containerID="4acf8c2134efecdbe32180d480d0b6a3ef66a084e541d61b31e6224be558c0ea"
Nov 03 21:10:18.834390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.834369    5533 scope.go:122] "RemoveContainer" containerID="4b949b2354806928fa7ac68d26b83d72bc87d758ffe164669ba41fdd99df8de8"
Nov 03 21:10:18.843221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:18.843193    5533 scope.go:122] "RemoveContainer" containerID="aab1f42505e9f7202680c3dc774357ce7c7514fe5845817bdfa9e60044ab5cec"
Nov 03 21:10:19.003872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.003838    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerName="payload-processing" containerID="cri-o://cd31aa70931df46fd1792172017b982b3c2ef1acc1ab3e992d61626d179b8102" gracePeriod=25
Nov 03 21:10:19.050556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.050486    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" podUID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" containerName="payload-processing" containerID="cri-o://c721fcf04fcf6deba2b600bde91640eaa6ab2cc4a4146009bbd2bde1bf40964f" gracePeriod=25
Nov 03 21:10:19.277759 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:10:19.277718    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:10:19.313267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.313223    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerID="cd31aa70931df46fd1792172017b982b3c2ef1acc1ab3e992d61626d179b8102" exitCode=0
Nov 03 21:10:19.313379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.313308    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" event={"ID":"8d288560-27b9-45a8-ba81-5cd9d1af6597","Type":"ContainerDied","Data":"cd31aa70931df46fd1792172017b982b3c2ef1acc1ab3e992d61626d179b8102"}
Nov 03 21:10:19.315570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.315545    5533 generic.go:356] "Generic (PLEG): container finished" podID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" containerID="c721fcf04fcf6deba2b600bde91640eaa6ab2cc4a4146009bbd2bde1bf40964f" exitCode=0
Nov 03 21:10:19.315705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.315618    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" event={"ID":"1998af65-866e-4c04-b0a8-ccd0ca6f494d","Type":"ContainerDied","Data":"c721fcf04fcf6deba2b600bde91640eaa6ab2cc4a4146009bbd2bde1bf40964f"}
Nov 03 21:10:19.333142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.333117    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:19.355157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.355136    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:10:19.496540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496511    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\") pod \"8d288560-27b9-45a8-ba81-5cd9d1af6597\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") "
Nov 03 21:10:19.496540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496548    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\") pod \"8d288560-27b9-45a8-ba81-5cd9d1af6597\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") "
Nov 03 21:10:19.496787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496585    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\") pod \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") "
Nov 03 21:10:19.496787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496626    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\") pod \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") "
Nov 03 21:10:19.496787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496696    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\") pod \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") "
Nov 03 21:10:19.496787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496740    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\") pod \"8d288560-27b9-45a8-ba81-5cd9d1af6597\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") "
Nov 03 21:10:19.496991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496789    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\") pod \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\" (UID: \"1998af65-866e-4c04-b0a8-ccd0ca6f494d\") "
Nov 03 21:10:19.496991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496841    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\") pod \"8d288560-27b9-45a8-ba81-5cd9d1af6597\" (UID: \"8d288560-27b9-45a8-ba81-5cd9d1af6597\") "
Nov 03 21:10:19.496991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.496924    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume" pod "8d288560-27b9-45a8-ba81-5cd9d1af6597" (UID: "8d288560-27b9-45a8-ba81-5cd9d1af6597"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:19.497238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.497219    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume" pod "8d288560-27b9-45a8-ba81-5cd9d1af6597" (UID: "8d288560-27b9-45a8-ba81-5cd9d1af6597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:19.497361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.497342    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.497416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.497369    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d288560-27b9-45a8-ba81-5cd9d1af6597-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.497727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.497697    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume" pod "1998af65-866e-4c04-b0a8-ccd0ca6f494d" (UID: "1998af65-866e-4c04-b0a8-ccd0ca6f494d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:19.498102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.498079    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume" pod "1998af65-866e-4c04-b0a8-ccd0ca6f494d" (UID: "1998af65-866e-4c04-b0a8-ccd0ca6f494d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:19.499936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.499907    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials" pod "8d288560-27b9-45a8-ba81-5cd9d1af6597" (UID: "8d288560-27b9-45a8-ba81-5cd9d1af6597"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.500359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.500342    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials" pod "1998af65-866e-4c04-b0a8-ccd0ca6f494d" (UID: "1998af65-866e-4c04-b0a8-ccd0ca6f494d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.500501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.500479    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt" pod "1998af65-866e-4c04-b0a8-ccd0ca6f494d" (UID: "1998af65-866e-4c04-b0a8-ccd0ca6f494d"). InnerVolumeSpecName "kube-api-access-tcgmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.501262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.501237    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78" pod "8d288560-27b9-45a8-ba81-5cd9d1af6597" (UID: "8d288560-27b9-45a8-ba81-5cd9d1af6597"). InnerVolumeSpecName "kube-api-access-j9h78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598830    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598866    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9h78\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-kube-api-access-j9h78\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598885    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgmt\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-kube-api-access-tcgmt\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598900    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d288560-27b9-45a8-ba81-5cd9d1af6597-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598915    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1998af65-866e-4c04-b0a8-ccd0ca6f494d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:19.599001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:19.598930    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1998af65-866e-4c04-b0a8-ccd0ca6f494d-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:20.212528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.212495    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:10:20.325098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.325031    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"
Nov 03 21:10:20.325250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.325031    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm" event={"ID":"8d288560-27b9-45a8-ba81-5cd9d1af6597","Type":"ContainerDied","Data":"ad0b08791ed40f2fd7827a013dd6d3623e161432be23e39f16d04e12822d0b6d"}
Nov 03 21:10:20.325250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.325180    5533 scope.go:122] "RemoveContainer" containerID="cd31aa70931df46fd1792172017b982b3c2ef1acc1ab3e992d61626d179b8102"
Nov 03 21:10:20.326521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.326502    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49"
Nov 03 21:10:20.326594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.326522    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59d447b455-z7x49" event={"ID":"1998af65-866e-4c04-b0a8-ccd0ca6f494d","Type":"ContainerDied","Data":"785314e4a28785a74bead9987caf18bff03f67e048d5895d4018da4ed667b2e0"}
Nov 03 21:10:20.336750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.336727    5533 scope.go:122] "RemoveContainer" containerID="c721fcf04fcf6deba2b600bde91640eaa6ab2cc4a4146009bbd2bde1bf40964f"
Nov 03 21:10:20.484413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.484377    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:20.531321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.531291    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd67c9464-4lhmm"]
Nov 03 21:10:20.633400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.633334    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:10:20.639850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:20.639827    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59d447b455-z7x49"]
Nov 03 21:10:22.501613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:22.501575    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" path="/var/lib/kubelet/pods/1998af65-866e-4c04-b0a8-ccd0ca6f494d/volumes"
Nov 03 21:10:22.502347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:22.502324    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" path="/var/lib/kubelet/pods/8d288560-27b9-45a8-ba81-5cd9d1af6597/volumes"
Nov 03 21:10:24.883134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883102    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:10:24.883760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883742    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" containerName="payload-processing"
Nov 03 21:10:24.883824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883762    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" containerName="payload-processing"
Nov 03 21:10:24.883824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883780    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerName="payload-processing"
Nov 03 21:10:24.883824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883786    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerName="payload-processing"
Nov 03 21:10:24.883925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883867    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1998af65-866e-4c04-b0a8-ccd0ca6f494d" containerName="payload-processing"
Nov 03 21:10:24.883925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.883877    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d288560-27b9-45a8-ba81-5cd9d1af6597" containerName="payload-processing"
Nov 03 21:10:24.889015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.888997    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909112    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909227    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909288    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909327    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909406    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909430    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.909495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.909457    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4gx\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:24.939678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.939629    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 21:10:24.939788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.939685    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 21:10:24.939788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.939724    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bh5bp\""
Nov 03 21:10:24.985418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.985391    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 21:10:24.991579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:24.991555    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:10:25.010795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.010771    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.010901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.010807    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.010901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.010849    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4gx\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.011000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.010983    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.011098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.011083    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.011158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.011113    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.011158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.011147    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.012053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.012030    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.013854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.013828    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.013939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.013857    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.014122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.014106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.014980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.014961    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.068951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.068920    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.069076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.069057    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4gx\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6d5tj\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.201533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.201505    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:25.626483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.626459    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:10:25.626974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:25.626948    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:10:26.391292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:26.391257    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" event={"ID":"14d18783-a004-48bb-b664-ef1edf7ea720","Type":"ContainerStarted","Data":"43da674823121ce56dff2aae733bd033bc0c73f6cd244a78c6935e2b6938c52e"}
Nov 03 21:10:30.137786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:30.137751    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:10:30.138198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:30.137824    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:10:30.429237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:30.429204    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" event={"ID":"14d18783-a004-48bb-b664-ef1edf7ea720","Type":"ContainerStarted","Data":"b4af9e549d4e48d008b3e1ca67b756664f5df460de0d4647f84ee7399b6a5dd7"}
Nov 03 21:10:30.429413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:30.429302    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:30.641121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:30.641065    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" podStartSLOduration=2.131083006 podStartE2EDuration="6.641052128s" totalImagesPullingTime="4.509969122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:30.590282474 +0000 UTC m=+1332.221595950" watchObservedRunningTime="2026-11-03 21:10:30.641052128 +0000 UTC m=+1332.272365605"
Nov 03 21:10:31.459979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:31.458070    5533 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6d5tj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:31.459979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:31.458137    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:34.456516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:10:34.456471    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:10:49.260405 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:10:49.260364    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:11:01.534238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:01.534164    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:11:11.894695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.894658    5533 generic.go:356] "Generic (PLEG): container finished" podID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerID="9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b" exitCode=0
Nov 03 21:11:11.895676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.895615    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerDied","Data":"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"}
Nov 03 21:11:11.896480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.896462    5533 scope.go:122] "RemoveContainer" containerID="9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"
Nov 03 21:11:11.903145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.903105    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:11.903270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.903246    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:11.903337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:11.903296    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:12.906194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:12.906161    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerStarted","Data":"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"}
Nov 03 21:11:13.718812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:13.718778    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:11:13.719665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:13.719628    5533 patch_prober.go:29] interesting pod/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 21:11:13.719774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:13.719694    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 21:11:14.718859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:14.718821    5533 patch_prober.go:29] interesting pod/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 21:11:14.719255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:14.718885    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 21:11:15.719190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:15.719150    5533 patch_prober.go:29] interesting pod/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 21:11:15.719591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:15.719210    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 21:11:16.718344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:16.718306    5533 patch_prober.go:29] interesting pod/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 21:11:16.718510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:16.718361    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 21:11:17.719241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:17.719203    5533 patch_prober.go:29] interesting pod/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 21:11:17.719656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:17.719264    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 21:11:18.142461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:18.142384    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:11:18.722622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:18.722595    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:11:18.900099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:18.900071    5533 scope.go:122] "RemoveContainer" containerID="151246e8a4be18e8e34b3373aa78255171f71aca71094fb3fa2b24704f21a218"
Nov 03 21:11:18.953168 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:18.953146    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:11:19.260381 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:11:19.260350    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:11:28.246452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:28.246414    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:11:38.538738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.538702    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:11:38.543014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.542989    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.592524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.592496    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3337df-openshift-default-dockercfg-r88h6\""
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.747678    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.747720    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.747750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.747874    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqrq\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.748052    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.748141    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.748289    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.748385    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.748657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.748525    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.761675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.757717    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:11:38.849852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.849756    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.849852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.849826    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.849920    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqrq\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.849975    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850032    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850084    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850142    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850222    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.850331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850316    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.851014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.850875    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.854742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.854682    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.855077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.855049    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.855229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.855120    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.856532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.855891    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.858565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.858534    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.859234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.859172    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.909803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.909772    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqrq\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:38.909803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:38.909792    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\") pod \"e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") " pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:39.162132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:39.162055    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:39.584503 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:11:39.584462    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675ce686_5167_422d_89d8_a6a1c15ef3de.slice/crio-eca31102370cff03670d6fda1c91904780345495a92a4a09fb039763790f7392 WatchSource:0}: Error finding container eca31102370cff03670d6fda1c91904780345495a92a4a09fb039763790f7392: Status 404 returned error can't find the container with id eca31102370cff03670d6fda1c91904780345495a92a4a09fb039763790f7392
Nov 03 21:11:39.587096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:39.587058    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:39.587214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:39.587125    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:39.587214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:39.587153    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:39.587214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:39.587070    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:11:40.162565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:40.162530    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerStarted","Data":"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"}
Nov 03 21:11:40.162840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:40.162586    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerStarted","Data":"eca31102370cff03670d6fda1c91904780345495a92a4a09fb039763790f7392"}
Nov 03 21:11:40.280884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:40.280834    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podStartSLOduration=2.280820602 podStartE2EDuration="2.280820602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:40.276596018 +0000 UTC m=+1401.907909495" watchObservedRunningTime="2026-11-03 21:11:40.280820602 +0000 UTC m=+1401.912134079"
Nov 03 21:11:41.162879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:41.162846    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:41.169908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:41.169881    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:41.172854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:41.172831    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:41.173515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:41.173494    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:11:46.095828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.095798    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:11:46.100274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.100254    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.152698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.152665    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.152836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.152748    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfvf\" (UniqueName: \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.153886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.153862    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:11:46.195988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.195859    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3e92e3\""
Nov 03 21:11:46.196413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.196388    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3e92e3-dockercfg-xhkgj\""
Nov 03 21:11:46.254543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.254503    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.254885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.254568    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfvf\" (UniqueName: \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.256543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.255891    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.318302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.318266    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfvf\" (UniqueName: \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\") pod \"payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.414544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.414462    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:46.793869 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:11:46.793833    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dee5365_7f79_49fb_9742_140a7ed1acee.slice/crio-0d42649146b54932ce65c2684c407cde4582c19bbd301dde9944a476ceeb418e WatchSource:0}: Error finding container 0d42649146b54932ce65c2684c407cde4582c19bbd301dde9944a476ceeb418e: Status 404 returned error can't find the container with id 0d42649146b54932ce65c2684c407cde4582c19bbd301dde9944a476ceeb418e
Nov 03 21:11:46.794251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:46.794228    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:11:47.236500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:47.236455    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" event={"ID":"0dee5365-7f79-49fb-9742-140a7ed1acee","Type":"ContainerStarted","Data":"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"}
Nov 03 21:11:47.236928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:47.236552    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" event={"ID":"0dee5365-7f79-49fb-9742-140a7ed1acee","Type":"ContainerStarted","Data":"0d42649146b54932ce65c2684c407cde4582c19bbd301dde9944a476ceeb418e"}
Nov 03 21:11:47.236928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:47.236766    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:11:47.348819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:47.348771    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" podStartSLOduration=1.348757564 podStartE2EDuration="1.348757564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:47.345622362 +0000 UTC m=+1408.976935838" watchObservedRunningTime="2026-11-03 21:11:47.348757564 +0000 UTC m=+1408.980071041"
Nov 03 21:11:49.266584 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:11:49.266540    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:11:51.491485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.491452    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:11:51.495957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.495936    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.547257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.547227    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3337df\""
Nov 03 21:11:51.568813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.568748    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nc7t\" (UniqueName: \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.569266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.569241    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.595042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.594228    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3337df-dockercfg-9j9gt\""
Nov 03 21:11:51.601907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.601883    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:11:51.671337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.671307    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc7t\" (UniqueName: \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.671466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.671430    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.672119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.672094    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.732291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.732257    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc7t\" (UniqueName: \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\") pod \"payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:51.811025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:51.810956    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:52.198470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:52.198440    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:11:52.287472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:52.287443    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" event={"ID":"2c5acd93-44cd-48bc-9ff3-859010e3c1cf","Type":"ContainerStarted","Data":"20f70cbe08eb720e60d58a236ce05f1f822cee44e929d5eacc102000c6efc2b5"}
Nov 03 21:11:53.297504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:53.297451    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" event={"ID":"2c5acd93-44cd-48bc-9ff3-859010e3c1cf","Type":"ContainerStarted","Data":"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"}
Nov 03 21:11:53.297997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:53.297559    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:11:53.409100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:53.409031    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" podStartSLOduration=2.409012166 podStartE2EDuration="2.409012166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:53.406882202 +0000 UTC m=+1415.038195682" watchObservedRunningTime="2026-11-03 21:11:53.409012166 +0000 UTC m=+1415.040325644"
Nov 03 21:11:58.248965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:11:58.248931    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:12:04.309241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:04.309213    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:12:19.261736 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:12:19.261687    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:12:21.973128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:21.973089    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:12:25.048541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:25.048503    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"]
Nov 03 21:12:25.095475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:25.095446    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67b311d-zbrbt"]
Nov 03 21:12:26.494584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:26.494552    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d36462a-0ffa-45f1-9052-5ffe36404afd" path="/var/lib/kubelet/pods/8d36462a-0ffa-45f1-9052-5ffe36404afd/volumes"
Nov 03 21:12:37.269213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.269178    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"]
Nov 03 21:12:37.274740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.274719    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.327747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.327721    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"]
Nov 03 21:12:37.368712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.368684    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:12:37.368830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.368734    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:12:37.417070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.417038    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsj9\" (UniqueName: \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.417201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.417159    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.417311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.417297    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.518871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.518840    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsj9\" (UniqueName: \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.519006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.518942    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.519089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.519073    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.519730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.519673    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.521286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.521265    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.577057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.577030    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsj9\" (UniqueName: \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\") pod \"maas-api-revoke-sub-063e17c2-fb5hb\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:37.590687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:37.590667    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:38.066565 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:12:38.066512    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611414be_1555_491d_b640_392e6c6c0f48.slice/crio-31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f WatchSource:0}: Error finding container 31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f: Status 404 returned error can't find the container with id 31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f
Nov 03 21:12:38.067889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:38.067857    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"]
Nov 03 21:12:38.716082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:38.716042    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb" event={"ID":"611414be-1555-491d-b640-392e6c6c0f48","Type":"ContainerStarted","Data":"3fd32f2783ebe36278edbdcf50a6a4513dd855cb7dbd3d98cb4239a4bd509fb0"}
Nov 03 21:12:38.716651 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:38.716610    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb" event={"ID":"611414be-1555-491d-b640-392e6c6c0f48","Type":"ContainerStarted","Data":"31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f"}
Nov 03 21:12:38.827226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:38.827173    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb" podStartSLOduration=1.82715026 podStartE2EDuration="1.82715026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:38.826408126 +0000 UTC m=+1460.457721605" watchObservedRunningTime="2026-11-03 21:12:38.82715026 +0000 UTC m=+1460.458463737"
Nov 03 21:12:39.725512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:39.725478    5533 generic.go:356] "Generic (PLEG): container finished" podID="611414be-1555-491d-b640-392e6c6c0f48" containerID="3fd32f2783ebe36278edbdcf50a6a4513dd855cb7dbd3d98cb4239a4bd509fb0" exitCode=0
Nov 03 21:12:39.725930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:39.725554    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb" event={"ID":"611414be-1555-491d-b640-392e6c6c0f48","Type":"ContainerDied","Data":"3fd32f2783ebe36278edbdcf50a6a4513dd855cb7dbd3d98cb4239a4bd509fb0"}
Nov 03 21:12:40.909482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.909453    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:40.993729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.993661    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\") pod \"611414be-1555-491d-b640-392e6c6c0f48\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") "
Nov 03 21:12:40.993865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.993836    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\" (UniqueName: \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\") pod \"611414be-1555-491d-b640-392e6c6c0f48\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") "
Nov 03 21:12:40.993908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.993897    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\") pod \"611414be-1555-491d-b640-392e6c6c0f48\" (UID: \"611414be-1555-491d-b640-392e6c6c0f48\") "
Nov 03 21:12:40.994196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.994168    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca" pod "611414be-1555-491d-b640-392e6c6c0f48" (UID: "611414be-1555-491d-b640-392e6c6c0f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:40.995745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.995719    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca" pod "611414be-1555-491d-b640-392e6c6c0f48" (UID: "611414be-1555-491d-b640-392e6c6c0f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:40.995897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:40.995881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9" pod "611414be-1555-491d-b640-392e6c6c0f48" (UID: "611414be-1555-491d-b640-392e6c6c0f48"). InnerVolumeSpecName "kube-api-access-fjsj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:41.071474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.071437    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"]
Nov 03 21:12:41.075476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.075453    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01908125-7228k"]
Nov 03 21:12:41.093069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.093042    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"]
Nov 03 21:12:41.093697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.093681    5533 cpu_manager.go:414] "containerMap: removing container" podUID="611414be-1555-491d-b640-392e6c6c0f48" containerName="revoke-keys"
Nov 03 21:12:41.093751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.093699    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611414be-1555-491d-b640-392e6c6c0f48" containerName="revoke-keys"
Nov 03 21:12:41.093822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.093813    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="611414be-1555-491d-b640-392e6c6c0f48" containerName="revoke-keys"
Nov 03 21:12:41.095344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.095302    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjsj9\" (UniqueName: \"kubernetes.io/projected/611414be-1555-491d-b640-392e6c6c0f48-kube-api-access-fjsj9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:41.095344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.095332    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611414be-1555-491d-b640-392e6c6c0f48-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:41.095516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.095349    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611414be-1555-491d-b640-392e6c6c0f48-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:41.101225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.101206    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.158197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.158171    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"]
Nov 03 21:12:41.196163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.196133    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5v6k\" (UniqueName: \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.196298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.196260    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.196360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.196329    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.297379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.297315    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.297481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.297412    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.297553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.297537    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5v6k\" (UniqueName: \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.298010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.297993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.299905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.299879    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.355205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.355186    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5v6k\" (UniqueName: \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.415960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.415940    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:41.742109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.742082    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"
Nov 03 21:12:41.742289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.742076    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb" event={"ID":"611414be-1555-491d-b640-392e6c6c0f48","Type":"ContainerDied","Data":"31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f"}
Nov 03 21:12:41.742289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.742227    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d28a149a6fd27d4d49f05619ef0aa61708b294adb8e4cc131fe9e35a09f08f"
Nov 03 21:12:41.797362 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:12:41.797331    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680026e7_3c4a_4df9_8149_1349daa416f7.slice/crio-ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c WatchSource:0}: Error finding container ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c: Status 404 returned error can't find the container with id ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c
Nov 03 21:12:41.797919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:41.797896    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"]
Nov 03 21:12:42.047404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.047335    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"]
Nov 03 21:12:42.052375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.052346    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8edece9f-4h788"]
Nov 03 21:12:42.494560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.494528    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4237e068-0c54-4243-a84c-59570b99134d" path="/var/lib/kubelet/pods/4237e068-0c54-4243-a84c-59570b99134d/volumes"
Nov 03 21:12:42.494998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.494984    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd72f58-5d6a-4201-95d0-d49a46e81d75" path="/var/lib/kubelet/pods/fdd72f58-5d6a-4201-95d0-d49a46e81d75/volumes"
Nov 03 21:12:42.751830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.751770    5533 generic.go:356] "Generic (PLEG): container finished" podID="680026e7-3c4a-4df9-8149-1349daa416f7" containerID="fb87c3f21e13243a01467f84097facf05a80b8c4f778fdf57a6babd9dd7a29fa" exitCode=0
Nov 03 21:12:42.751945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.751846    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs" event={"ID":"680026e7-3c4a-4df9-8149-1349daa416f7","Type":"ContainerDied","Data":"fb87c3f21e13243a01467f84097facf05a80b8c4f778fdf57a6babd9dd7a29fa"}
Nov 03 21:12:42.751945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:42.751899    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs" event={"ID":"680026e7-3c4a-4df9-8149-1349daa416f7","Type":"ContainerStarted","Data":"ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c"}
Nov 03 21:12:43.926287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:43.926261    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:44.033520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.033451    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\") pod \"680026e7-3c4a-4df9-8149-1349daa416f7\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") "
Nov 03 21:12:44.033520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.033489    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\" (UniqueName: \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\") pod \"680026e7-3c4a-4df9-8149-1349daa416f7\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") "
Nov 03 21:12:44.033759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.033528    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\") pod \"680026e7-3c4a-4df9-8149-1349daa416f7\" (UID: \"680026e7-3c4a-4df9-8149-1349daa416f7\") "
Nov 03 21:12:44.033846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.033821    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca" pod "680026e7-3c4a-4df9-8149-1349daa416f7" (UID: "680026e7-3c4a-4df9-8149-1349daa416f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:44.034032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.034017    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:44.035674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.035646    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k" pod "680026e7-3c4a-4df9-8149-1349daa416f7" (UID: "680026e7-3c4a-4df9-8149-1349daa416f7"). InnerVolumeSpecName "kube-api-access-s5v6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:44.035783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.035716    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca" pod "680026e7-3c4a-4df9-8149-1349daa416f7" (UID: "680026e7-3c4a-4df9-8149-1349daa416f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:44.135155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.135112    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5v6k\" (UniqueName: \"kubernetes.io/projected/680026e7-3c4a-4df9-8149-1349daa416f7-kube-api-access-s5v6k\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:44.135155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.135150    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680026e7-3c4a-4df9-8149-1349daa416f7-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:44.768075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.768000    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"
Nov 03 21:12:44.768075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.768009    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs" event={"ID":"680026e7-3c4a-4df9-8149-1349daa416f7","Type":"ContainerDied","Data":"ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c"}
Nov 03 21:12:44.768248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:44.768083    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2026668e033e9d572dc4d66a765c83587885ff8a736239e29384cc9e0fc08c"
Nov 03 21:12:49.261098 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:12:49.261064    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:12:52.047076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:52.047039    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"]
Nov 03 21:12:52.093458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:52.093425    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c0f24-mchtd"]
Nov 03 21:12:52.494466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:52.494434    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f542f4b0-039c-40e0-802c-e67f5f474704" path="/var/lib/kubelet/pods/f542f4b0-039c-40e0-802c-e67f5f474704/volumes"
Nov 03 21:12:52.965169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:52.965132    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:12:56.047284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:56.047244    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"]
Nov 03 21:12:56.094810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:56.094779    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9c5b73-tslcg"]
Nov 03 21:12:56.504174 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:56.504142    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867e50a4-9417-4e14-bca7-61d4ee99b280" path="/var/lib/kubelet/pods/867e50a4-9417-4e14-bca7-61d4ee99b280/volumes"
Nov 03 21:12:57.965715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:57.965672    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" podUID="3caf9cd6-f035-4717-bb49-81b25587c968" containerName="payload-pre-processing" containerID="cri-o://38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c" gracePeriod=25
Nov 03 21:12:58.243422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.243395    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:12:58.309866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.309840    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\" (UniqueName: \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\") pod \"3caf9cd6-f035-4717-bb49-81b25587c968\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") "
Nov 03 21:12:58.310029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.309907    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\") pod \"3caf9cd6-f035-4717-bb49-81b25587c968\" (UID: \"3caf9cd6-f035-4717-bb49-81b25587c968\") "
Nov 03 21:12:58.310247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.310227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume" pod "3caf9cd6-f035-4717-bb49-81b25587c968" (UID: "3caf9cd6-f035-4717-bb49-81b25587c968"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:58.310389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.310375    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3caf9cd6-f035-4717-bb49-81b25587c968-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:58.311965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.311943    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8" pod "3caf9cd6-f035-4717-bb49-81b25587c968" (UID: "3caf9cd6-f035-4717-bb49-81b25587c968"). InnerVolumeSpecName "kube-api-access-vjjw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:58.411938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.411906    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjw8\" (UniqueName: \"kubernetes.io/projected/3caf9cd6-f035-4717-bb49-81b25587c968-kube-api-access-vjjw8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:58.878788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.878692    5533 generic.go:356] "Generic (PLEG): container finished" podID="3caf9cd6-f035-4717-bb49-81b25587c968" containerID="38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c" exitCode=0
Nov 03 21:12:58.878788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.878736    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"
Nov 03 21:12:58.879009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.878782    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" event={"ID":"3caf9cd6-f035-4717-bb49-81b25587c968","Type":"ContainerDied","Data":"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"}
Nov 03 21:12:58.879009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.878838    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66" event={"ID":"3caf9cd6-f035-4717-bb49-81b25587c968","Type":"ContainerDied","Data":"a10aac484ce76c7edf0a74921cfbeab15f0f814286ae2b8f2d49bac0c4d6165d"}
Nov 03 21:12:58.879009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.878860    5533 scope.go:122] "RemoveContainer" containerID="38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"
Nov 03 21:12:58.890348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.890318    5533 scope.go:122] "RemoveContainer" containerID="38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"
Nov 03 21:12:58.890695 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:12:58.890669    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c\": container with ID starting with 38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c not found: ID does not exist" containerID="38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"
Nov 03 21:12:58.890795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:58.890705    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c"} err="failed to get container status \"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c\": rpc error: code = NotFound desc = could not find container \"38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c\": container with ID starting with 38e20de664544b221172eda13c0276de465538c229427550f80c3358f5f4a52c not found: ID does not exist"
Nov 03 21:12:59.036448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:59.036420    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:12:59.082692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:59.082662    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-32f6e43c-6bbcf95cc5-t4g66"]
Nov 03 21:12:59.602316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:12:59.602269    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:13:00.506807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:00.506771    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3caf9cd6-f035-4717-bb49-81b25587c968" path="/var/lib/kubelet/pods/3caf9cd6-f035-4717-bb49-81b25587c968/volumes"
Nov 03 21:13:09.051673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:09.051620    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"]
Nov 03 21:13:09.096454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:09.096420    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f123d07-w2wlw"]
Nov 03 21:13:10.496001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:10.495951    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6917e9e9-01fc-41b7-b175-3bae5636ab29" path="/var/lib/kubelet/pods/6917e9e9-01fc-41b7-b175-3bae5636ab29/volumes"
Nov 03 21:13:12.046673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:12.046624    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"]
Nov 03 21:13:12.093350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:12.093309    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8511881a-pqhpw"]
Nov 03 21:13:12.493539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:12.493507    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ece2f3-f999-4ccb-bc82-b9e722450c49" path="/var/lib/kubelet/pods/a2ece2f3-f999-4ccb-bc82-b9e722450c49/volumes"
Nov 03 21:13:17.561587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.561549    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:13:17.681786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.681751    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:13:17.836012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.835941    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"]
Nov 03 21:13:17.836570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836555    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3caf9cd6-f035-4717-bb49-81b25587c968" containerName="payload-pre-processing"
Nov 03 21:13:17.836679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836570    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3caf9cd6-f035-4717-bb49-81b25587c968" containerName="payload-pre-processing"
Nov 03 21:13:17.836679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836595    5533 cpu_manager.go:414] "containerMap: removing container" podUID="680026e7-3c4a-4df9-8149-1349daa416f7" containerName="revoke-keys"
Nov 03 21:13:17.836679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836600    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680026e7-3c4a-4df9-8149-1349daa416f7" containerName="revoke-keys"
Nov 03 21:13:17.836824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836762    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3caf9cd6-f035-4717-bb49-81b25587c968" containerName="payload-pre-processing"
Nov 03 21:13:17.836824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.836777    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="680026e7-3c4a-4df9-8149-1349daa416f7" containerName="revoke-keys"
Nov 03 21:13:17.841331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.841314    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:17.899821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.899792    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"]
Nov 03 21:13:17.900734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.900712    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:13:17.900991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.900975    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:13:17.950029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.949996    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:17.950164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.950070    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:17.950206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:17.950187    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68vt\" (UniqueName: \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.051626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.051591    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.051884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.051711    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68vt\" (UniqueName: \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.051968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.051949    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.052609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.052588    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.054016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.053989    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.109773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.109706    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68vt\" (UniqueName: \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\") pod \"maas-api-revoke-sub-1c80f6b5-wcp9n\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.155130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.155106    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:18.536674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.536504    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"]
Nov 03 21:13:18.537017 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:13:18.536981    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0218c44_799a_4d03_8bbb_e36eaf5ecd35.slice/crio-fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e WatchSource:0}: Error finding container fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e: Status 404 returned error can't find the container with id fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e
Nov 03 21:13:18.700222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.700194    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:13:18.705591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.705563    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:13:18.713328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.713303    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:13:18.717250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:18.717228    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:13:19.071658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:19.071064    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n" event={"ID":"e0218c44-799a-4d03-8bbb-e36eaf5ecd35","Type":"ContainerStarted","Data":"50b95816784d2112efa22b7c237bc913b74e756a9f826d8a331fcd33044959a4"}
Nov 03 21:13:19.071658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:19.071184    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n" event={"ID":"e0218c44-799a-4d03-8bbb-e36eaf5ecd35","Type":"ContainerStarted","Data":"fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e"}
Nov 03 21:13:19.187558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:19.187506    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n" podStartSLOduration=2.187489298 podStartE2EDuration="2.187489298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:19.182999979 +0000 UTC m=+1500.814313456" watchObservedRunningTime="2026-11-03 21:13:19.187489298 +0000 UTC m=+1500.818802778"
Nov 03 21:13:19.264803 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:13:19.264758    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:13:20.082998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:20.082965    5533 generic.go:356] "Generic (PLEG): container finished" podID="e0218c44-799a-4d03-8bbb-e36eaf5ecd35" containerID="50b95816784d2112efa22b7c237bc913b74e756a9f826d8a331fcd33044959a4" exitCode=0
Nov 03 21:13:20.083445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:20.083049    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n" event={"ID":"e0218c44-799a-4d03-8bbb-e36eaf5ecd35","Type":"ContainerDied","Data":"50b95816784d2112efa22b7c237bc913b74e756a9f826d8a331fcd33044959a4"}
Nov 03 21:13:21.256088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.256066    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:21.304921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.304897    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\") pod \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") "
Nov 03 21:13:21.305115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.305102    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\") pod \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") "
Nov 03 21:13:21.305157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.305131    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\" (UniqueName: \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\") pod \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\" (UID: \"e0218c44-799a-4d03-8bbb-e36eaf5ecd35\") "
Nov 03 21:13:21.305222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.305203    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca" pod "e0218c44-799a-4d03-8bbb-e36eaf5ecd35" (UID: "e0218c44-799a-4d03-8bbb-e36eaf5ecd35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:21.305488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.305473    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:21.307200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.307177    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca" pod "e0218c44-799a-4d03-8bbb-e36eaf5ecd35" (UID: "e0218c44-799a-4d03-8bbb-e36eaf5ecd35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:13:21.307296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.307265    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt" pod "e0218c44-799a-4d03-8bbb-e36eaf5ecd35" (UID: "e0218c44-799a-4d03-8bbb-e36eaf5ecd35"). InnerVolumeSpecName "kube-api-access-z68vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:21.406816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.406748    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:21.406816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:21.406772    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68vt\" (UniqueName: \"kubernetes.io/projected/e0218c44-799a-4d03-8bbb-e36eaf5ecd35-kube-api-access-z68vt\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:22.098886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.098850    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n" event={"ID":"e0218c44-799a-4d03-8bbb-e36eaf5ecd35","Type":"ContainerDied","Data":"fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e"}
Nov 03 21:13:22.099033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.098926    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa6ffe97bad265d9b4431cec0bb8f6cfa4e87ea4e1aaa3e28aa857bff901684e"
Nov 03 21:13:22.099033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.098865    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"
Nov 03 21:13:22.571257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.569303    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" podUID="4b668561-2741-4627-a775-08de62fbf073" containerName="payload-processing" containerID="cri-o://7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd" gracePeriod=25
Nov 03 21:13:22.683382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.682698    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" podUID="78befc1d-adf6-499b-a498-aee6791cf275" containerName="payload-pre-processing" containerID="cri-o://86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1" gracePeriod=25
Nov 03 21:13:22.893221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:22.893191    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:13:23.002343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.002316    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:13:23.026941 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.026909    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\") pod \"4b668561-2741-4627-a775-08de62fbf073\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") "
Nov 03 21:13:23.027117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.027043    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\" (UniqueName: \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\") pod \"4b668561-2741-4627-a775-08de62fbf073\" (UID: \"4b668561-2741-4627-a775-08de62fbf073\") "
Nov 03 21:13:23.027386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.027358    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume" pod "4b668561-2741-4627-a775-08de62fbf073" (UID: "4b668561-2741-4627-a775-08de62fbf073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:23.027855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.027833    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b668561-2741-4627-a775-08de62fbf073-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:23.029427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.029398    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr" pod "4b668561-2741-4627-a775-08de62fbf073" (UID: "4b668561-2741-4627-a775-08de62fbf073"). InnerVolumeSpecName "kube-api-access-zt8wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:23.058647 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.058574    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"]
Nov 03 21:13:23.065527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.065500    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba17051-d428z"]
Nov 03 21:13:23.108259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.108230    5533 generic.go:356] "Generic (PLEG): container finished" podID="4b668561-2741-4627-a775-08de62fbf073" containerID="7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd" exitCode=0
Nov 03 21:13:23.108366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.108270    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"
Nov 03 21:13:23.108366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.108313    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" event={"ID":"4b668561-2741-4627-a775-08de62fbf073","Type":"ContainerDied","Data":"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"}
Nov 03 21:13:23.108495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.108368    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx" event={"ID":"4b668561-2741-4627-a775-08de62fbf073","Type":"ContainerDied","Data":"49feada38265c21d02aa18185b758870d6560bb4b7bec2f9b7927101935b8dc2"}
Nov 03 21:13:23.108495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.108396    5533 scope.go:122] "RemoveContainer" containerID="7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"
Nov 03 21:13:23.109962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.109920    5533 generic.go:356] "Generic (PLEG): container finished" podID="78befc1d-adf6-499b-a498-aee6791cf275" containerID="86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1" exitCode=0
Nov 03 21:13:23.109962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.109959    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"
Nov 03 21:13:23.110135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.109956    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" event={"ID":"78befc1d-adf6-499b-a498-aee6791cf275","Type":"ContainerDied","Data":"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"}
Nov 03 21:13:23.110135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.110010    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm" event={"ID":"78befc1d-adf6-499b-a498-aee6791cf275","Type":"ContainerDied","Data":"bd4b2d3e0e2cd42927e97d95bb1e2f9a5d202cb9f64fbebca45fabbb0dbce1ba"}
Nov 03 21:13:23.120674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.120621    5533 scope.go:122] "RemoveContainer" containerID="7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"
Nov 03 21:13:23.121044 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:13:23.120981    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd\": container with ID starting with 7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd not found: ID does not exist" containerID="7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"
Nov 03 21:13:23.121176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.121039    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd"} err="failed to get container status \"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd\": rpc error: code = NotFound desc = could not find container \"7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd\": container with ID starting with 7fc94cb8102919f4792170c08dee740a3eb350cad8cc5461a0d9059f2fe5c7cd not found: ID does not exist"
Nov 03 21:13:23.121176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.121063    5533 scope.go:122] "RemoveContainer" containerID="86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"
Nov 03 21:13:23.134049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.134025    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\" (UniqueName: \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\") pod \"78befc1d-adf6-499b-a498-aee6791cf275\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") "
Nov 03 21:13:23.134158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.134093    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\") pod \"78befc1d-adf6-499b-a498-aee6791cf275\" (UID: \"78befc1d-adf6-499b-a498-aee6791cf275\") "
Nov 03 21:13:23.134516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.134490    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume" pod "78befc1d-adf6-499b-a498-aee6791cf275" (UID: "78befc1d-adf6-499b-a498-aee6791cf275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:23.134735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.134715    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt8wr\" (UniqueName: \"kubernetes.io/projected/4b668561-2741-4627-a775-08de62fbf073-kube-api-access-zt8wr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:23.134823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.134741    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78befc1d-adf6-499b-a498-aee6791cf275-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:23.136187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.136101    5533 scope.go:122] "RemoveContainer" containerID="86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"
Nov 03 21:13:23.136351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.136330    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6" pod "78befc1d-adf6-499b-a498-aee6791cf275" (UID: "78befc1d-adf6-499b-a498-aee6791cf275"). InnerVolumeSpecName "kube-api-access-vltc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:23.136705 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:13:23.136676    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1\": container with ID starting with 86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1 not found: ID does not exist" containerID="86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"
Nov 03 21:13:23.136817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.136710    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1"} err="failed to get container status \"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1\": rpc error: code = NotFound desc = could not find container \"86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1\": container with ID starting with 86eabe5ed11ba53bb07676c62012e2459787a1939817e76d5488112cdac07da1 not found: ID does not exist"
Nov 03 21:13:23.229144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.229114    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:13:23.233062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.233039    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-145cc5-76544964cb-5zdqx"]
Nov 03 21:13:23.235460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.235436    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltc6\" (UniqueName: \"kubernetes.io/projected/78befc1d-adf6-499b-a498-aee6791cf275-kube-api-access-vltc6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:23.535862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.535826    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:13:23.541803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:23.541770    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-145cc5-78b865d679-m9gvm"]
Nov 03 21:13:24.495178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:24.495139    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48055325-b2bf-4366-9ca9-6e193241f991" path="/var/lib/kubelet/pods/48055325-b2bf-4366-9ca9-6e193241f991/volumes"
Nov 03 21:13:24.495593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:24.495578    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b668561-2741-4627-a775-08de62fbf073" path="/var/lib/kubelet/pods/4b668561-2741-4627-a775-08de62fbf073/volumes"
Nov 03 21:13:24.495996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:24.495983    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78befc1d-adf6-499b-a498-aee6791cf275" path="/var/lib/kubelet/pods/78befc1d-adf6-499b-a498-aee6791cf275/volumes"
Nov 03 21:13:26.047969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.047936    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"]
Nov 03 21:13:26.096513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.096477    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9429373e-6vddm"]
Nov 03 21:13:26.513669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.511661    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c452b2-814d-4045-86b2-5253c3ff772e" path="/var/lib/kubelet/pods/06c452b2-814d-4045-86b2-5253c3ff772e/volumes"
Nov 03 21:13:26.513669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.512112    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:13:26.514375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.514349    5533 cpu_manager.go:414] "containerMap: removing container" podUID="78befc1d-adf6-499b-a498-aee6791cf275" containerName="payload-pre-processing"
Nov 03 21:13:26.514853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.514833    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78befc1d-adf6-499b-a498-aee6791cf275" containerName="payload-pre-processing"
Nov 03 21:13:26.515267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515249    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e0218c44-799a-4d03-8bbb-e36eaf5ecd35" containerName="revoke-keys"
Nov 03 21:13:26.515371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515360    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0218c44-799a-4d03-8bbb-e36eaf5ecd35" containerName="revoke-keys"
Nov 03 21:13:26.515516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515505    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4b668561-2741-4627-a775-08de62fbf073" containerName="payload-processing"
Nov 03 21:13:26.515586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515577    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b668561-2741-4627-a775-08de62fbf073" containerName="payload-processing"
Nov 03 21:13:26.515818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515803    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b668561-2741-4627-a775-08de62fbf073" containerName="payload-processing"
Nov 03 21:13:26.515912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515903    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="78befc1d-adf6-499b-a498-aee6791cf275" containerName="payload-pre-processing"
Nov 03 21:13:26.515979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.515972    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0218c44-799a-4d03-8bbb-e36eaf5ecd35" containerName="revoke-keys"
Nov 03 21:13:26.527656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.523303    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.608517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.573760    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-22600c-openshift-default-dockercfg-9zvmw\""
Nov 03 21:13:26.709109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709054    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whnl\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709137    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709169    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709418    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709442    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709465    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709489    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.709654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.709567    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.714468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.712527    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.810832    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.810880    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.810905    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.810933    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.810999    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.811100    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whnl\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.811205    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.811247    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.811385    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.811753    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.812576    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.813258    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.813706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.813508    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.814473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.813800    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.825686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.821618    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.826324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.826296    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.876447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.870535    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.876447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.871013    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whnl\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\") pod \"e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") " pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:26.882710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:26.880055    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:27.313010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:27.312980    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:13:27.313410 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:13:27.313282    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f17222_e41f_4f14_9917_57c949467fe6.slice/crio-101b78ff48fad05727700c349833c54fbe69f9a33f70749d81f3b686a8b54c98 WatchSource:0}: Error finding container 101b78ff48fad05727700c349833c54fbe69f9a33f70749d81f3b686a8b54c98: Status 404 returned error can't find the container with id 101b78ff48fad05727700c349833c54fbe69f9a33f70749d81f3b686a8b54c98
Nov 03 21:13:27.315943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:27.315916    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:27.316029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:27.315975    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:27.316029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:27.316001    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:28.152101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:28.152062    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerStarted","Data":"146a9d322d3a2bcf0932555b3a54189c8c179209ae32ed28fa008bcf6003c9c1"}
Nov 03 21:13:28.152285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:28.152142    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerStarted","Data":"101b78ff48fad05727700c349833c54fbe69f9a33f70749d81f3b686a8b54c98"}
Nov 03 21:13:28.315764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:28.315704    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podStartSLOduration=2.315688975 podStartE2EDuration="2.315688975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:28.312248842 +0000 UTC m=+1509.943562335" watchObservedRunningTime="2026-11-03 21:13:28.315688975 +0000 UTC m=+1509.947002451"
Nov 03 21:13:28.880563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:28.880532    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:28.885581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:28.885557    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:29.219658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:29.210767    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:29.219658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:29.214875    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:13:37.048116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.048081    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"]
Nov 03 21:13:37.097372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.097330    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266267b1-c76b4"]
Nov 03 21:13:37.585660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.582331    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"]
Nov 03 21:13:37.595657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.595441    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.639368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.639336    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"]
Nov 03 21:13:37.647577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.647550    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.647724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.647675    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.647795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.647778    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.675877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.675843    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:13:37.695563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.676257    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:13:37.748997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.748967    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.749144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.749023    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.749144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.749098    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.749623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.749598    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.790190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.790142    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.831778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.831553    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\") pod \"maas-api-revoke-sub-f44806f0-nctt5\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:37.917494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:37.917412    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:38.334955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:38.334927    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"]
Nov 03 21:13:38.335494 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:13:38.335467    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41be67c_0b5c_43f1_ac30_3e1e51d817b3.slice/crio-d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058 WatchSource:0}: Error finding container d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058: Status 404 returned error can't find the container with id d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058
Nov 03 21:13:38.496364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:38.496283    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ebe29d-2c4c-44ff-9904-992d297f1f89" path="/var/lib/kubelet/pods/36ebe29d-2c4c-44ff-9904-992d297f1f89/volumes"
Nov 03 21:13:38.917066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:38.917035    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:13:38.921318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:38.921298    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.101692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.101552    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:13:39.109220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.109190    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0f8726-openshift-default-dockercfg-9wxwc\""
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137165    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7dgq\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137240    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137322    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137362    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137388    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137471    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137520    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137577    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.140686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.137759    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.238750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.238716    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.238931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.238804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.238931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.238919    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7dgq\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.238978    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239006    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239037    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239061    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239114    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239181    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.239747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.239721    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.240039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.240014    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.240107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.240046    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.240304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.240277    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.240304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.240293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.242397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.242370    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.243429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.243400    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.296466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.296436    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.297040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.297010    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7dgq\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\") pod \"e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") " pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.331654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.331605    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5" event={"ID":"c41be67c-0b5c-43f1-ac30-3e1e51d817b3","Type":"ContainerStarted","Data":"33d363c8892ed85f4d135c8a9b241133f2236f0aee24645a383286ec52f04c19"}
Nov 03 21:13:39.331835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.331686    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5" event={"ID":"c41be67c-0b5c-43f1-ac30-3e1e51d817b3","Type":"ContainerStarted","Data":"d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058"}
Nov 03 21:13:39.454206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.454132    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5" podStartSLOduration=2.454113805 podStartE2EDuration="2.454113805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:39.448988763 +0000 UTC m=+1521.080302241" watchObservedRunningTime="2026-11-03 21:13:39.454113805 +0000 UTC m=+1521.085427288"
Nov 03 21:13:39.540657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.535971    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:39.797225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.792937    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:13:39.800226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.799167    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:39.854372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.851996    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:13:39.881062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.880741    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:39.881062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.880866    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8bc\" (UniqueName: \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:39.936716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.933972    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-22600c\""
Nov 03 21:13:39.936716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.934156    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-22600c-dockercfg-szgg5\""
Nov 03 21:13:39.982750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.982710    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:39.982951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.982816    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8bc\" (UniqueName: \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:39.983979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:39.983928    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:40.041242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.041206    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8bc\" (UniqueName: \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\") pod \"payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:40.073491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.073453    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:13:40.073961 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:13:40.073935    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8161d5_9550_40b0_acc2_96b7040adb07.slice/crio-30f37dd9abc5f79cc8cfccd92618dd57492fe6f50edc62cd23a64f1be4c0bead WatchSource:0}: Error finding container 30f37dd9abc5f79cc8cfccd92618dd57492fe6f50edc62cd23a64f1be4c0bead: Status 404 returned error can't find the container with id 30f37dd9abc5f79cc8cfccd92618dd57492fe6f50edc62cd23a64f1be4c0bead
Nov 03 21:13:40.076971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.076936    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:40.077092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.076997    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:40.077092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.077024    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:40.126671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.124996    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:40.341227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.341149    5533 generic.go:356] "Generic (PLEG): container finished" podID="c41be67c-0b5c-43f1-ac30-3e1e51d817b3" containerID="33d363c8892ed85f4d135c8a9b241133f2236f0aee24645a383286ec52f04c19" exitCode=0
Nov 03 21:13:40.341370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.341226    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5" event={"ID":"c41be67c-0b5c-43f1-ac30-3e1e51d817b3","Type":"ContainerDied","Data":"33d363c8892ed85f4d135c8a9b241133f2236f0aee24645a383286ec52f04c19"}
Nov 03 21:13:40.343162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.343129    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerStarted","Data":"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"}
Nov 03 21:13:40.343302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.343179    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerStarted","Data":"30f37dd9abc5f79cc8cfccd92618dd57492fe6f50edc62cd23a64f1be4c0bead"}
Nov 03 21:13:40.531581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.531546    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:13:40.531581 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:13:40.531555    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cc166d_4a1a_4be6_bd24_0628b404b697.slice/crio-07e7946de5f2978bace054bc7c2de830c03bb402277e3b67a0035c8b2f970b35 WatchSource:0}: Error finding container 07e7946de5f2978bace054bc7c2de830c03bb402277e3b67a0035c8b2f970b35: Status 404 returned error can't find the container with id 07e7946de5f2978bace054bc7c2de830c03bb402277e3b67a0035c8b2f970b35
Nov 03 21:13:40.536853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.536828    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:40.568107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:40.568063    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podStartSLOduration=2.568050019 podStartE2EDuration="2.568050019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:40.563621097 +0000 UTC m=+1522.194934574" watchObservedRunningTime="2026-11-03 21:13:40.568050019 +0000 UTC m=+1522.199363560"
Nov 03 21:13:41.355202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.355150    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" event={"ID":"f1cc166d-4a1a-4be6-bd24-0628b404b697","Type":"ContainerStarted","Data":"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"}
Nov 03 21:13:41.355403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.355212    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" event={"ID":"f1cc166d-4a1a-4be6-bd24-0628b404b697","Type":"ContainerStarted","Data":"07e7946de5f2978bace054bc7c2de830c03bb402277e3b67a0035c8b2f970b35"}
Nov 03 21:13:41.475242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.475175    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" podStartSLOduration=2.4751503440000002 podStartE2EDuration="2.475150344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:41.466089847 +0000 UTC m=+1523.097403325" watchObservedRunningTime="2026-11-03 21:13:41.475150344 +0000 UTC m=+1523.106463825"
Nov 03 21:13:41.533579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.533547    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:41.543177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.543143    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:41.618938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.618859    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\") pod \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") "
Nov 03 21:13:41.619097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.618943    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\") pod \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") "
Nov 03 21:13:41.619191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.619171    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\") pod \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\" (UID: \"c41be67c-0b5c-43f1-ac30-3e1e51d817b3\") "
Nov 03 21:13:41.625922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.622347    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca" pod "c41be67c-0b5c-43f1-ac30-3e1e51d817b3" (UID: "c41be67c-0b5c-43f1-ac30-3e1e51d817b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:13:41.626320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.626295    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca" pod "c41be67c-0b5c-43f1-ac30-3e1e51d817b3" (UID: "c41be67c-0b5c-43f1-ac30-3e1e51d817b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:13:41.630951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.630912    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7" pod "c41be67c-0b5c-43f1-ac30-3e1e51d817b3" (UID: "c41be67c-0b5c-43f1-ac30-3e1e51d817b3"). InnerVolumeSpecName "kube-api-access-wtsv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:41.722059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.722020    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:41.722059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.722062    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:41.722313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:41.722078    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/c41be67c-0b5c-43f1-ac30-3e1e51d817b3-kube-api-access-wtsv7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:42.050728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.050691    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"]
Nov 03 21:13:42.054352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.054321    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75eae842-wmtl4"]
Nov 03 21:13:42.214769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.214737    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:42.214969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.214794    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:42.367042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.366966    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"
Nov 03 21:13:42.367042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.367001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5" event={"ID":"c41be67c-0b5c-43f1-ac30-3e1e51d817b3","Type":"ContainerDied","Data":"d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058"}
Nov 03 21:13:42.367220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.367049    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5bbc02e7d58522500253d02c4f00773d18ad6f1a192591fcc5d14608e856058"
Nov 03 21:13:42.367668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.367648    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:42.367854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.367832    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:42.368489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.368462    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:13:42.496464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:42.496422    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3025edb8-641e-4581-b9b1-4205b9c857f9" path="/var/lib/kubelet/pods/3025edb8-641e-4581-b9b1-4205b9c857f9/volumes"
Nov 03 21:13:43.377359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.377326    5533 generic.go:356] "Generic (PLEG): container finished" podID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerID="f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16" exitCode=0
Nov 03 21:13:43.377893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.377421    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerDied","Data":"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"}
Nov 03 21:13:43.378057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.378039    5533 scope.go:122] "RemoveContainer" containerID="f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"
Nov 03 21:13:43.379712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.379679    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:43.379841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.379764    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:43.379841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:43.379812    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:13:44.388193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:44.388154    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerStarted","Data":"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"}
Nov 03 21:13:45.167714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:45.166745    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:13:45.167714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:45.166883    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 21:13:45.167714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:45.166926    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 21:13:46.168660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:46.163694    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 21:13:46.168660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:46.163769    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 21:13:47.169960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:47.169923    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 21:13:47.170544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:47.170511    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 21:13:48.163675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:48.163622    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 21:13:48.163952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:48.163921    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 21:13:49.203659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:49.194049    5533 patch_prober.go:29] interesting pod/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 21:13:49.203659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:49.194109    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 21:13:49.288904 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:13:49.288861    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:13:50.167586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:50.167556    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:13:50.168097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:50.168072    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:13:50.168710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:50.168680    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:13:52.047908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.047873    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"]
Nov 03 21:13:52.094374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.094345    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"]
Nov 03 21:13:52.097526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.097513    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e84f5a8-bql8v"]
Nov 03 21:13:52.101827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.101799    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd957444-rwq4g"]
Nov 03 21:13:52.494337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.494309    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489547ee-81ea-49ee-bc50-63d063aa4e2c" path="/var/lib/kubelet/pods/489547ee-81ea-49ee-bc50-63d063aa4e2c/volumes"
Nov 03 21:13:52.494745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:52.494731    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a27319-b624-45e4-b778-0e5112e40432" path="/var/lib/kubelet/pods/82a27319-b624-45e4-b778-0e5112e40432/volumes"
Nov 03 21:13:53.046553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:53.046515    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"]
Nov 03 21:13:53.050537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:53.050508    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21d8b9c-x6lq6"]
Nov 03 21:13:53.381495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:53.381416    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:13:54.494364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:13:54.494330    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654fd73d-4cf1-41d1-b0a2-ee56728b1cfe" path="/var/lib/kubelet/pods/654fd73d-4cf1-41d1-b0a2-ee56728b1cfe/volumes"
Nov 03 21:14:02.147136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:02.147050    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:14:03.046741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:03.046703    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"]
Nov 03 21:14:03.103621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:03.103526    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4053930a-2fr4x"]
Nov 03 21:14:04.494245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:04.494213    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c7df40-1342-4db6-b005-a00a07e2524b" path="/var/lib/kubelet/pods/d7c7df40-1342-4db6-b005-a00a07e2524b/volumes"
Nov 03 21:14:06.229752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:06.229683    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:14:06.230190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:06.229999    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy" containerID="cri-o://5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041" gracePeriod=30
Nov 03 21:14:08.047401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:08.047366    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"]
Nov 03 21:14:08.094031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:08.093997    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03af4e7-ktk4n"]
Nov 03 21:14:08.175682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:08.174284    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:14:08.175682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:08.174527    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy" containerID="cri-o://368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba" gracePeriod=30
Nov 03 21:14:08.499569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:08.499524    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f" path="/var/lib/kubelet/pods/5b16a8c1-7b0d-4b99-ae73-b3779c91ec1f/volumes"
Nov 03 21:14:11.501180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.501121    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:14:11.559046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559017    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559110    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559143    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559178    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559204    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559244    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559280    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559307    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:11.559582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559413    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:11.559582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559430    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559491    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\") pod \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\" (UID: \"8ad77ce8-9e3f-4630-916e-d6dafd597c2a\") "
Nov 03 21:14:11.559771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559606    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:11.559771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559695    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:11.559928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.559856    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:11.560363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.560095    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.560363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.560120    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.560363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.560133    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.560363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.560146    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.560363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.560160    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.562036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.562007    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "kube-api-access-txjhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:11.562036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.562020    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:11.562212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.562153    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:11.562979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.562957    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token" pod "8ad77ce8-9e3f-4630-916e-d6dafd597c2a" (UID: "8ad77ce8-9e3f-4630-916e-d6dafd597c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:11.626208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.626179    5533 generic.go:356] "Generic (PLEG): container finished" podID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerID="5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041" exitCode=0
Nov 03 21:14:11.626376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.626222    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"
Nov 03 21:14:11.626376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.626261    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" event={"ID":"8ad77ce8-9e3f-4630-916e-d6dafd597c2a","Type":"ContainerDied","Data":"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"}
Nov 03 21:14:11.626376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.626314    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87" event={"ID":"8ad77ce8-9e3f-4630-916e-d6dafd597c2a","Type":"ContainerDied","Data":"432cea5eeed322558caea09facf6ce09ed83f9c482239034710c246a368b7570"}
Nov 03 21:14:11.626376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.626338    5533 scope.go:122] "RemoveContainer" containerID="5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"
Nov 03 21:14:11.643316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.643302    5533 scope.go:122] "RemoveContainer" containerID="5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"
Nov 03 21:14:11.643585 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:11.643562    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041\": container with ID starting with 5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041 not found: ID does not exist" containerID="5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"
Nov 03 21:14:11.643689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.643588    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041"} err="failed to get container status \"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041\": rpc error: code = NotFound desc = could not find container \"5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041\": container with ID starting with 5218ed6577a665a211c0e6c007a4354c21193b301513d6f0e1193ae97d638041 not found: ID does not exist"
Nov 03 21:14:11.661679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.661588    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.661679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.661615    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjhv\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-kube-api-access-txjhv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.661679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.661630    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.661679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.661658    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad77ce8-9e3f-4630-916e-d6dafd597c2a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:11.768692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.768601    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:14:11.773218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:11.773191    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-9mx87"]
Nov 03 21:14:12.495738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:12.495707    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" path="/var/lib/kubelet/pods/8ad77ce8-9e3f-4630-916e-d6dafd597c2a/volumes"
Nov 03 21:14:13.446111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.446052    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:14:13.486308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486284    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486322    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486356    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486406    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486485    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486515    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486577    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:13.486671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486612    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486728    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486821    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\") pod \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\" (UID: \"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6\") "
Nov 03 21:14:13.486904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486862    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:13.486946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486900    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:13.487002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.486980    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:13.487167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487139    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:13.487388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487366    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.487388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487389    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.487388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487399    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.487593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487407    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.487593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.487420    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.488708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.488686    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:13.488954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.488937    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:13.489085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.489053    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:13.489085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.489083    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg" pod "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" (UID: "dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6"). InnerVolumeSpecName "kube-api-access-gd5kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:13.588921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.588888    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.588921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.588926    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5kg\" (UniqueName: \"kubernetes.io/projected/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-kube-api-access-gd5kg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.589098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.588943    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.589098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.588958    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:13.645061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.645032    5533 generic.go:356] "Generic (PLEG): container finished" podID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerID="368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba" exitCode=0
Nov 03 21:14:13.645225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.645102    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"
Nov 03 21:14:13.645225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.645115    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerDied","Data":"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"}
Nov 03 21:14:13.645225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.645177    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs" event={"ID":"dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6","Type":"ContainerDied","Data":"8eafc05f6c05065199b976d969d218de9d7397525c75923852f5a3bf49c5327b"}
Nov 03 21:14:13.645225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.645210    5533 scope.go:122] "RemoveContainer" containerID="368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"
Nov 03 21:14:13.662665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.662615    5533 scope.go:122] "RemoveContainer" containerID="9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"
Nov 03 21:14:13.672580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.672559    5533 scope.go:122] "RemoveContainer" containerID="368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"
Nov 03 21:14:13.672868 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:13.672849    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba\": container with ID starting with 368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba not found: ID does not exist" containerID="368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"
Nov 03 21:14:13.672923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.672877    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba"} err="failed to get container status \"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba\": rpc error: code = NotFound desc = could not find container \"368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba\": container with ID starting with 368b373f7dca27f2c917368d701d9ccbedf20c7b33bb3a17bfc51369ef3bf9ba not found: ID does not exist"
Nov 03 21:14:13.672923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.672895    5533 scope.go:122] "RemoveContainer" containerID="9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"
Nov 03 21:14:13.673100 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:13.673080    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b\": container with ID starting with 9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b not found: ID does not exist" containerID="9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"
Nov 03 21:14:13.673136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.673108    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b"} err="failed to get container status \"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b\": rpc error: code = NotFound desc = could not find container \"9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b\": container with ID starting with 9eda5b4c708dcae173ad21e06755e0ed491821db7f4056abd64c53ac36fc158b not found: ID does not exist"
Nov 03 21:14:13.808565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.808514    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:14:13.814178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:13.814157    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6dacc6-openshift-default-5ff849fddc-mwcrs"]
Nov 03 21:14:14.494137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:14.494109    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" path="/var/lib/kubelet/pods/dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6/volumes"
Nov 03 21:14:16.166701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:16.166670    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:14:18.974559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:18.974530    5533 scope.go:122] "RemoveContainer" containerID="baaf8deaf613e791828ace0b1aa9ce612c6968865d89cb3184ac03c3749f3b42"
Nov 03 21:14:18.984861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:18.984841    5533 scope.go:122] "RemoveContainer" containerID="b77cf22278b4f6ef5f89e8d8e8f803547510757a3c151d7b543f4cf7e63348c7"
Nov 03 21:14:18.993962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:18.993941    5533 scope.go:122] "RemoveContainer" containerID="501449f4dab3a32e6695adfa358abb8cb381ff6a701a7088b0a4fa400c9f5c14"
Nov 03 21:14:19.002727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.002706    5533 scope.go:122] "RemoveContainer" containerID="c1dbf034c8180e1448bbee5733ffebb7949800ebc93758d7c29e3d31837cb553"
Nov 03 21:14:19.011024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.011003    5533 scope.go:122] "RemoveContainer" containerID="c645d41a4c81a96d296e43869fe8fc866272f9865983c3e8b8777698148c22bf"
Nov 03 21:14:19.019148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.019131    5533 scope.go:122] "RemoveContainer" containerID="57143f482a8d553078c6c32a0d367fc3ab35ef81d32a6afdcc569d511922d0cf"
Nov 03 21:14:19.027274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.027256    5533 scope.go:122] "RemoveContainer" containerID="7dc590ffb2398082f8b4d26080644491916c689f36f29305b07a4bed5e95dcd8"
Nov 03 21:14:19.048359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.048323    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"]
Nov 03 21:14:19.094336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:19.094315    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bed7d42a-s6wvn"]
Nov 03 21:14:19.260347 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:19.260268    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:14:20.496579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:20.496543    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12b0762-623c-4f19-bed8-e1f640db0f11" path="/var/lib/kubelet/pods/f12b0762-623c-4f19-bed8-e1f640db0f11/volumes"
Nov 03 21:14:21.167543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.167489    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" podUID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" containerName="payload-pre-processing" containerID="cri-o://d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631" gracePeriod=25
Nov 03 21:14:21.431944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.431923    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:14:21.495202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.495179    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\") pod \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") "
Nov 03 21:14:21.495328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.495241    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\" (UniqueName: \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\") pod \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\" (UID: \"2c5acd93-44cd-48bc-9ff3-859010e3c1cf\") "
Nov 03 21:14:21.495530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.495507    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume" pod "2c5acd93-44cd-48bc-9ff3-859010e3c1cf" (UID: "2c5acd93-44cd-48bc-9ff3-859010e3c1cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:21.495755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.495740    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:21.497261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.497236    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t" pod "2c5acd93-44cd-48bc-9ff3-859010e3c1cf" (UID: "2c5acd93-44cd-48bc-9ff3-859010e3c1cf"). InnerVolumeSpecName "kube-api-access-7nc7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:21.597830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.597795    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nc7t\" (UniqueName: \"kubernetes.io/projected/2c5acd93-44cd-48bc-9ff3-859010e3c1cf-kube-api-access-7nc7t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:21.714845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.714764    5533 generic.go:356] "Generic (PLEG): container finished" podID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" containerID="d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631" exitCode=0
Nov 03 21:14:21.714845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.714814    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"
Nov 03 21:14:21.714845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.714830    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" event={"ID":"2c5acd93-44cd-48bc-9ff3-859010e3c1cf","Type":"ContainerDied","Data":"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"}
Nov 03 21:14:21.715072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.714885    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p" event={"ID":"2c5acd93-44cd-48bc-9ff3-859010e3c1cf","Type":"ContainerDied","Data":"20f70cbe08eb720e60d58a236ce05f1f822cee44e929d5eacc102000c6efc2b5"}
Nov 03 21:14:21.715072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.714908    5533 scope.go:122] "RemoveContainer" containerID="d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"
Nov 03 21:14:21.725522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.725503    5533 scope.go:122] "RemoveContainer" containerID="d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"
Nov 03 21:14:21.725821 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:21.725802    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631\": container with ID starting with d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631 not found: ID does not exist" containerID="d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"
Nov 03 21:14:21.725903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.725835    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631"} err="failed to get container status \"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631\": rpc error: code = NotFound desc = could not find container \"d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631\": container with ID starting with d4552ebef4735febac6b3fccc16b9280a8405c2712c438fa025b83d603120631 not found: ID does not exist"
Nov 03 21:14:21.872725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.872698    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:14:21.876600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:21.876575    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3337df-75b5bf78d7-zwd8p"]
Nov 03 21:14:22.493957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:22.493926    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" path="/var/lib/kubelet/pods/2c5acd93-44cd-48bc-9ff3-859010e3c1cf/volumes"
Nov 03 21:14:29.923155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:29.923122    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"]
Nov 03 21:14:29.969395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:29.969355    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-sznjs"]
Nov 03 21:14:30.498428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:30.498389    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680026e7-3c4a-4df9-8149-1349daa416f7" path="/var/lib/kubelet/pods/680026e7-3c4a-4df9-8149-1349daa416f7/volumes"
Nov 03 21:14:39.087230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:39.087189    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"]
Nov 03 21:14:39.090433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:39.090405    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b68627-8rfkf"]
Nov 03 21:14:40.495900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:40.495865    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a4f880-b67e-4aef-9b16-07c3c79e1f59" path="/var/lib/kubelet/pods/c8a4f880-b67e-4aef-9b16-07c3c79e1f59/volumes"
Nov 03 21:14:42.687536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.687499    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:14:42.689805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689779    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.689805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689804    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689821    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c41be67c-0b5c-43f1-ac30-3e1e51d817b3" containerName="revoke-keys"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689828    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41be67c-0b5c-43f1-ac30-3e1e51d817b3" containerName="revoke-keys"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689839    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689848    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689905    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" containerName="payload-pre-processing"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689912    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" containerName="payload-pre-processing"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689932    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy"
Nov 03 21:14:42.690012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.689940    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy"
Nov 03 21:14:42.690428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.690057    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5acd93-44cd-48bc-9ff3-859010e3c1cf" containerName="payload-pre-processing"
Nov 03 21:14:42.690428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.690072    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad77ce8-9e3f-4630-916e-d6dafd597c2a" containerName="istio-proxy"
Nov 03 21:14:42.690428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.690082    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.690428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.690092    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41be67c-0b5c-43f1-ac30-3e1e51d817b3" containerName="revoke-keys"
Nov 03 21:14:42.690428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.690100    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec14b1d-ebe4-4d84-bc2c-f2b6822d33b6" containerName="istio-proxy"
Nov 03 21:14:42.694597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.694573    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.771172 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.771131    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:14:42.789121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.789088    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-32f6e43c-dockercfg-qkg6h\""
Nov 03 21:14:42.789492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.789460    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-32f6e43c\""
Nov 03 21:14:42.835859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.835817    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljbl\" (UniqueName: \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.836053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.835983    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.937095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.937054    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.937308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.937274    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljbl\" (UniqueName: \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.937890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.937836    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:42.999321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:42.999255    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljbl\" (UniqueName: \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\") pod \"payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") " pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:43.011927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:43.011884    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:43.390823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:43.390745    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:14:43.397029 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:14:43.396989    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e0fb5a_8902_4418_ad6f_198c0e3ed2c8.slice/crio-8577d1b8e313bd9b5491205fdaf8e483d30ca9d31c08001a91f8d5837ced1507 WatchSource:0}: Error finding container 8577d1b8e313bd9b5491205fdaf8e483d30ca9d31c08001a91f8d5837ced1507: Status 404 returned error can't find the container with id 8577d1b8e313bd9b5491205fdaf8e483d30ca9d31c08001a91f8d5837ced1507
Nov 03 21:14:43.941042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:43.940228    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" event={"ID":"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8","Type":"ContainerStarted","Data":"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"}
Nov 03 21:14:43.941042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:43.940299    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" event={"ID":"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8","Type":"ContainerStarted","Data":"8577d1b8e313bd9b5491205fdaf8e483d30ca9d31c08001a91f8d5837ced1507"}
Nov 03 21:14:43.941042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:43.940559    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:44.054880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:44.054811    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" podStartSLOduration=2.054792767 podStartE2EDuration="2.054792767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:44.049225827 +0000 UTC m=+1585.680539302" watchObservedRunningTime="2026-11-03 21:14:44.054792767 +0000 UTC m=+1585.686106245"
Nov 03 21:14:45.270660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.268437    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:14:45.270660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.268532    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:14:45.958522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.958493    5533 generic.go:356] "Generic (PLEG): container finished" podID="f4f17222-e41f-4f14-9917-57c949467fe6" containerID="146a9d322d3a2bcf0932555b3a54189c8c179209ae32ed28fa008bcf6003c9c1" exitCode=0
Nov 03 21:14:45.958705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.958565    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerDied","Data":"146a9d322d3a2bcf0932555b3a54189c8c179209ae32ed28fa008bcf6003c9c1"}
Nov 03 21:14:45.958979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.958962    5533 scope.go:122] "RemoveContainer" containerID="146a9d322d3a2bcf0932555b3a54189c8c179209ae32ed28fa008bcf6003c9c1"
Nov 03 21:14:45.960436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.960412    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:14:45.960497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.960476    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:14:45.960533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:45.960510    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:14:46.992317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:46.991836    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerStarted","Data":"f6f1b78828babc6d91c62457fd78f766affe2da74bc4681c7b9c793ec41daf88"}
Nov 03 21:14:47.092455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:47.092417    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"]
Nov 03 21:14:47.095876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:47.095850    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d5cfe1c-z64c9"]
Nov 03 21:14:47.882661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:47.881990    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:14:47.882661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:47.882113    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 21:14:47.882661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:47.882156    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 21:14:48.494572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:48.494538    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8513bd7f-3494-4cac-a2f2-c15c38d86f57" path="/var/lib/kubelet/pods/8513bd7f-3494-4cac-a2f2-c15c38d86f57/volumes"
Nov 03 21:14:48.881137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:48.881065    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 21:14:48.881137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:48.881120    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 21:14:49.279553 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:49.279502    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:14:49.881487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:49.881456    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 21:14:49.881963 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:49.881509    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 21:14:50.047670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:50.047631    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"]
Nov 03 21:14:50.094796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:50.094755    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766d02c0-fbpsz"]
Nov 03 21:14:50.499313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:50.499277    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e381d43c-8f06-494e-813e-44dfd11b988a" path="/var/lib/kubelet/pods/e381d43c-8f06-494e-813e-44dfd11b988a/volumes"
Nov 03 21:14:50.881474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:50.881394    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 21:14:50.881474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:50.881451    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 21:14:51.235436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.235400    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:14:51.235678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.235650    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy" containerID="cri-o://9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd" gracePeriod=30
Nov 03 21:14:51.288718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.288680    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:14:51.289178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.289143    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" containerID="cri-o://a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea" gracePeriod=30
Nov 03 21:14:51.294323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.294292    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:14:51.294552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.294527    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy" containerID="cri-o://f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623" gracePeriod=30
Nov 03 21:14:51.333376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.333351    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:14:51.333567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.333532    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy" containerID="cri-o://c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738" gracePeriod=30
Nov 03 21:14:51.881287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.881253    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 21:14:51.881456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:51.881315    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 21:14:52.885655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:52.885553    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:14:52.886168 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:52.886150    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:14:52.886971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:52.886948    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:14:54.952613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:54.952583    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:14:55.822775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:55.822742    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:14:55.822953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:55.822792    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:14:56.577354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.577322    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:14:56.626895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.626859    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:14:56.683207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683180    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683226    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683284    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683336    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683366    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683400    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683435    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683480    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683540    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683584    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683688    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.683821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683721    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.683821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683783    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683838    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683843    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683944    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683961    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.683969    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.684038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684022    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.684295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684091    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\") pod \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\" (UID: \"fb5e727e-11a6-43cd-90b3-b8fa803b0aca\") "
Nov 03 21:14:56.684295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684182    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\") pod \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\" (UID: \"0d793b54-bfb8-4b36-8595-c2bd76c7152b\") "
Nov 03 21:14:56.684401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684288    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.684624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684478    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684804    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684883    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684904    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684919    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684933    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.685049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.684948    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.685573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.685495    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.686042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686018    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.686141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686066    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:56.686313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686241    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.686313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686266    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.686991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686915    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "kube-api-access-hlwvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.686991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.686956    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.687278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.687253    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "kube-api-access-m78p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.687385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.687270    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.687986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.687968    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:56.688090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.688068    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token" pod "0d793b54-bfb8-4b36-8595-c2bd76c7152b" (UID: "0d793b54-bfb8-4b36-8595-c2bd76c7152b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.688761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.688743    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy" pod "fb5e727e-11a6-43cd-90b3-b8fa803b0aca" (UID: "fb5e727e-11a6-43cd-90b3-b8fa803b0aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.709836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.709819    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:14:56.712561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.712542    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:14:56.785935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.785860    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.785935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.785926    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.786102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.785975    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.786102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786006    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.786102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786036    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.786223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786125    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.786223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786164    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.786903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786580    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.786903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786587    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:56.786903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.786724    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.787548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.787521    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.787930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.787909    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.788037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.787956    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788022    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788226    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.788454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788430    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:56.788535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788471    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.788596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788561    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788598    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.788681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788657    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788709    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788759    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.788933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788832    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.788933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788852    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.788933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788883    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\") pod \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\" (UID: \"8472e6eb-de5e-494a-a0f7-f1d98d326d9b\") "
Nov 03 21:14:56.789083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.788932    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\") pod \"e2032c52-efe2-4944-91c8-f21325df3149\" (UID: \"e2032c52-efe2-4944-91c8-f21325df3149\") "
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789273    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789301    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789477    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789514    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789563    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789583    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789597    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789610    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789627    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwvr\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-kube-api-access-hlwvr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789661    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789675    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789688    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789703    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789716    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789730    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789744    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78p6\" (UniqueName: \"kubernetes.io/projected/0d793b54-bfb8-4b36-8595-c2bd76c7152b-kube-api-access-m78p6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789757    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.789761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789771    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789786    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789800    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789814    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789828    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789841    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789854    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5e727e-11a6-43cd-90b3-b8fa803b0aca-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789867    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2032c52-efe2-4944-91c8-f21325df3149-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789880    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.789895    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d793b54-bfb8-4b36-8595-c2bd76c7152b-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.790919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.790003    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:14:56.791332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.791088    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:56.791332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.791192    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.791332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.791228    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:14:56.791718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.791615    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc" pod "e2032c52-efe2-4944-91c8-f21325df3149" (UID: "e2032c52-efe2-4944-91c8-f21325df3149"). InnerVolumeSpecName "kube-api-access-fktsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.792093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.792076    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw" pod "8472e6eb-de5e-494a-a0f7-f1d98d326d9b" (UID: "8472e6eb-de5e-494a-a0f7-f1d98d326d9b"). InnerVolumeSpecName "kube-api-access-pjdxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:56.890946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890915    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdxw\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-kube-api-access-pjdxw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.890946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890944    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2032c52-efe2-4944-91c8-f21325df3149-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890956    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890965    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890975    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2032c52-efe2-4944-91c8-f21325df3149-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890983    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.890991    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8472e6eb-de5e-494a-a0f7-f1d98d326d9b-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:56.891142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:56.891000    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktsc\" (UniqueName: \"kubernetes.io/projected/e2032c52-efe2-4944-91c8-f21325df3149-kube-api-access-fktsc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:57.074472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.074393    5533 generic.go:356] "Generic (PLEG): container finished" podID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerID="c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738" exitCode=0
Nov 03 21:14:57.074472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.074438    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"
Nov 03 21:14:57.074670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.074470    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" event={"ID":"8472e6eb-de5e-494a-a0f7-f1d98d326d9b","Type":"ContainerDied","Data":"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"}
Nov 03 21:14:57.074670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.074522    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt" event={"ID":"8472e6eb-de5e-494a-a0f7-f1d98d326d9b","Type":"ContainerDied","Data":"49d11e5880c857eb197d1ed85cf724ae421e7206361e90f35838ffd9c9e20187"}
Nov 03 21:14:57.074670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.074553    5533 scope.go:122] "RemoveContainer" containerID="c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"
Nov 03 21:14:57.076001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.075978    5533 generic.go:356] "Generic (PLEG): container finished" podID="e2032c52-efe2-4944-91c8-f21325df3149" containerID="f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623" exitCode=0
Nov 03 21:14:57.076130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.076038    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"
Nov 03 21:14:57.076130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.076059    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" event={"ID":"e2032c52-efe2-4944-91c8-f21325df3149","Type":"ContainerDied","Data":"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"}
Nov 03 21:14:57.076130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.076110    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8" event={"ID":"e2032c52-efe2-4944-91c8-f21325df3149","Type":"ContainerDied","Data":"d1746d212b89272c3fa83182b8cf7b8180e8b70b42e58837d8680ff0663c14c2"}
Nov 03 21:14:57.077932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.077912    5533 generic.go:356] "Generic (PLEG): container finished" podID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerID="a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea" exitCode=0
Nov 03 21:14:57.078026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.077933    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerDied","Data":"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"}
Nov 03 21:14:57.078026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.077967    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"
Nov 03 21:14:57.078026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.077981    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4" event={"ID":"0d793b54-bfb8-4b36-8595-c2bd76c7152b","Type":"ContainerDied","Data":"7c72b1cf0d187c255ecfe4698072a5559191bfcd919f4ac08bfff6928fb6cc1c"}
Nov 03 21:14:57.079829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.079807    5533 generic.go:356] "Generic (PLEG): container finished" podID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerID="9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd" exitCode=0
Nov 03 21:14:57.079942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.079844    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"
Nov 03 21:14:57.079942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.079859    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" event={"ID":"fb5e727e-11a6-43cd-90b3-b8fa803b0aca","Type":"ContainerDied","Data":"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"}
Nov 03 21:14:57.079942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.079897    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692" event={"ID":"fb5e727e-11a6-43cd-90b3-b8fa803b0aca","Type":"ContainerDied","Data":"e6d8971c0586287c8d7f9dfde529335b0faf3791f6aed5e91a966ced833749ca"}
Nov 03 21:14:57.094039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.094020    5533 scope.go:122] "RemoveContainer" containerID="c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"
Nov 03 21:14:57.094367 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:57.094342    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738\": container with ID starting with c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738 not found: ID does not exist" containerID="c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"
Nov 03 21:14:57.094453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.094381    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738"} err="failed to get container status \"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738\": rpc error: code = NotFound desc = could not find container \"c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738\": container with ID starting with c178a7dee466dd90f5623a80f203125106fc210647c9cb7b874fa9026306d738 not found: ID does not exist"
Nov 03 21:14:57.094453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.094409    5533 scope.go:122] "RemoveContainer" containerID="f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"
Nov 03 21:14:57.119499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.119474    5533 scope.go:122] "RemoveContainer" containerID="f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"
Nov 03 21:14:57.119904 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:57.119879    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623\": container with ID starting with f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623 not found: ID does not exist" containerID="f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"
Nov 03 21:14:57.119994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.119916    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623"} err="failed to get container status \"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623\": rpc error: code = NotFound desc = could not find container \"f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623\": container with ID starting with f498c41701a4e480d1c204eb8dd07f6a7fc4997374d23e482fec7aa194990623 not found: ID does not exist"
Nov 03 21:14:57.119994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.119941    5533 scope.go:122] "RemoveContainer" containerID="a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"
Nov 03 21:14:57.131388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.131366    5533 scope.go:122] "RemoveContainer" containerID="0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"
Nov 03 21:14:57.140734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.140716    5533 scope.go:122] "RemoveContainer" containerID="a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"
Nov 03 21:14:57.141005 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:57.140981    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea\": container with ID starting with a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea not found: ID does not exist" containerID="a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"
Nov 03 21:14:57.141083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.141012    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea"} err="failed to get container status \"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea\": rpc error: code = NotFound desc = could not find container \"a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea\": container with ID starting with a35b04a112b7c9c8b8a3b0f4814999c8aebbe017f2cfeaecea9a26b8faa21bea not found: ID does not exist"
Nov 03 21:14:57.141083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.141027    5533 scope.go:122] "RemoveContainer" containerID="0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"
Nov 03 21:14:57.141321 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:57.141296    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967\": container with ID starting with 0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967 not found: ID does not exist" containerID="0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"
Nov 03 21:14:57.141379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.141334    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967"} err="failed to get container status \"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967\": rpc error: code = NotFound desc = could not find container \"0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967\": container with ID starting with 0a0ba47274b3857c2f399fa7ffbe1ffaa1868f691ddcaa9e81b58cb92745f967 not found: ID does not exist"
Nov 03 21:14:57.141379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.141356    5533 scope.go:122] "RemoveContainer" containerID="9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"
Nov 03 21:14:57.150723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.150703    5533 scope.go:122] "RemoveContainer" containerID="9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"
Nov 03 21:14:57.150976 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:14:57.150958    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd\": container with ID starting with 9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd not found: ID does not exist" containerID="9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"
Nov 03 21:14:57.151036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.150981    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd"} err="failed to get container status \"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd\": rpc error: code = NotFound desc = could not find container \"9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd\": container with ID starting with 9f4345f487cfc4089ae3a90f3e8db88e715ff05aa24d9cb6b2350fd9a30801dd not found: ID does not exist"
Nov 03 21:14:57.391629 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.391500    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:14:57.396673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.396615    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-6x4rt"]
Nov 03 21:14:57.500720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.498539    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:14:57.502242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.502218    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-grwp8"]
Nov 03 21:14:57.614183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.614143    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:14:57.616251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.616223    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-5kdd4"]
Nov 03 21:14:57.720752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.720723    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:14:57.725141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:57.725113    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-86692"]
Nov 03 21:14:58.495684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:58.495656    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" path="/var/lib/kubelet/pods/0d793b54-bfb8-4b36-8595-c2bd76c7152b/volumes"
Nov 03 21:14:58.496168 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:58.496145    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" path="/var/lib/kubelet/pods/8472e6eb-de5e-494a-a0f7-f1d98d326d9b/volumes"
Nov 03 21:14:58.496573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:58.496561    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2032c52-efe2-4944-91c8-f21325df3149" path="/var/lib/kubelet/pods/e2032c52-efe2-4944-91c8-f21325df3149/volumes"
Nov 03 21:14:58.497297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:14:58.497281    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" path="/var/lib/kubelet/pods/fb5e727e-11a6-43cd-90b3-b8fa803b0aca/volumes"
Nov 03 21:15:00.165943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.165915    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:00.166472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166456    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166474    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166489    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166495    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166502    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166507    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166515    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166520    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166527    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166532    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166673    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166683    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5e727e-11a6-43cd-90b3-b8fa803b0aca" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166688    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d793b54-bfb8-4b36-8595-c2bd76c7152b" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166695    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2032c52-efe2-4944-91c8-f21325df3149" containerName="istio-proxy"
Nov 03 21:15:00.166739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.166701    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8472e6eb-de5e-494a-a0f7-f1d98d326d9b" containerName="istio-proxy"
Nov 03 21:15:00.171701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.171682    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:00.171833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.171815    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.175697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.175668    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:00.175818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.175797    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.180160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.180138    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.210756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.210736    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:00.214369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.214354    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:00.214484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.214468    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.218228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.218210    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:00.218340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.218323    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.220842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.220824    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:15:00.221764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.221744    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:00.221860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.221843    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.225402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.225381    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.227850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.227828    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.227953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.227879    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.228045 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.228029    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwjx\" (UniqueName: \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.228121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.228083    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.228192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.228174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnmh\" (UniqueName: \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.228253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.228221    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfb6\" (UniqueName: \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.260700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.260568    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:00.268656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.266917    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:00.305507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.305480    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:00.312358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.312325    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:00.317156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.317129    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:00.321658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.321614    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:00.329924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.329898    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n8p\" (UniqueName: \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.330000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.329932    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.330000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.329968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnmh\" (UniqueName: \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.330075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330048    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.330115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330097    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xhz\" (UniqueName: \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.330155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330137    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.330193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330179    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfb6\" (UniqueName: \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.330270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330252    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.330328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330307    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bxz\" (UniqueName: \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.330392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330378    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.330428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330408    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.330468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330446    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.330504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330495    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.330545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.330536    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwjx\" (UniqueName: \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.331385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.331357    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.331476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.331392    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.331540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.331483    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.390858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.390837    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfb6\" (UniqueName: \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.390983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.390896    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwjx\" (UniqueName: \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\") pod \"maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.390983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.390957    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnmh\" (UniqueName: \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\") pod \"maas-api-key-cleanup-29895675-4mq8z\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.434095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434058    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.434311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434170    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n8p\" (UniqueName: \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.434311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434242    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.434311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434280    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xhz\" (UniqueName: \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.434484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434321    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.434484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434423    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bxz\" (UniqueName: \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.434580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434519    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.434668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434599    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.434991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.434959    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.435533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.435507    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.435807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.435784    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.435953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.435931    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.460582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.460547    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:00.486575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.486549    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:00.492690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.492651    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.493341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.493315    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bxz\" (UniqueName: \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\") pod \"maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.493909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.493882    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n8p\" (UniqueName: \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\") pod \"maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.494260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.494235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xhz\" (UniqueName: \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\") pod \"maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.495157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.495134    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:00.503270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.503247    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:00.529026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.528996    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:00.536420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.536395    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:00.545024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.544849    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:00.551399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.550951    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:00.897575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.897395    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:00.948366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.948282    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:00.951723 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:00.951697    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93555532_d28d_4b30_96a7_d217e8474cc5.slice/crio-3b2b47e9557647003f9b3b4c381673cd5af4d7d3233467e94935d24e9cecc28a WatchSource:0}: Error finding container 3b2b47e9557647003f9b3b4c381673cd5af4d7d3233467e94935d24e9cecc28a: Status 404 returned error can't find the container with id 3b2b47e9557647003f9b3b4c381673cd5af4d7d3233467e94935d24e9cecc28a
Nov 03 21:15:00.951837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:00.951739    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:00.955350 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:00.955305    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fd72d9_c37b_45ec_a377_3dab0d6e029a.slice/crio-01325354da81c8b86d202b9e12830830f817d7d4120f800b03bd1d7eaf7ac7e9 WatchSource:0}: Error finding container 01325354da81c8b86d202b9e12830830f817d7d4120f800b03bd1d7eaf7ac7e9: Status 404 returned error can't find the container with id 01325354da81c8b86d202b9e12830830f817d7d4120f800b03bd1d7eaf7ac7e9
Nov 03 21:15:01.001229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.001201    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:01.001555 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:01.001532    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e536b92_58e4_420c_bbaf_e23ca396a048.slice/crio-0b5eec2a118a861bffc66a3b2db400eae65b451d7fd567dc174e06f334d427fa WatchSource:0}: Error finding container 0b5eec2a118a861bffc66a3b2db400eae65b451d7fd567dc174e06f334d427fa: Status 404 returned error can't find the container with id 0b5eec2a118a861bffc66a3b2db400eae65b451d7fd567dc174e06f334d427fa
Nov 03 21:15:01.012584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.010713    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:01.018150 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:01.018122    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d01da5c_b90d_4c0f_8ecb_6d2ec02e1063.slice/crio-b2b2fb0484c6fd8ee57c13a719654754492436a23431cb8d8729c5057a55a9c6 WatchSource:0}: Error finding container b2b2fb0484c6fd8ee57c13a719654754492436a23431cb8d8729c5057a55a9c6: Status 404 returned error can't find the container with id b2b2fb0484c6fd8ee57c13a719654754492436a23431cb8d8729c5057a55a9c6
Nov 03 21:15:01.047260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.047225    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"]
Nov 03 21:15:01.050717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.050692    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c957e5-jjzh2"]
Nov 03 21:15:01.058692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.058668    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:01.085812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.085793    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:01.116573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.116544    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerStarted","Data":"01325354da81c8b86d202b9e12830830f817d7d4120f800b03bd1d7eaf7ac7e9"}
Nov 03 21:15:01.118157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.118128    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerStarted","Data":"dcec2133e1250896c2be0922f638a270b5b39fcb7260d4d3c1b7001797890cc1"}
Nov 03 21:15:01.119457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.119435    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" event={"ID":"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063","Type":"ContainerStarted","Data":"b2b2fb0484c6fd8ee57c13a719654754492436a23431cb8d8729c5057a55a9c6"}
Nov 03 21:15:01.120687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.120667    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerStarted","Data":"0b5eec2a118a861bffc66a3b2db400eae65b451d7fd567dc174e06f334d427fa"}
Nov 03 21:15:01.121955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:01.121936    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerStarted","Data":"3b2b47e9557647003f9b3b4c381673cd5af4d7d3233467e94935d24e9cecc28a"}
Nov 03 21:15:01.160673 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:01.160543    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd885b08_7c24_491a_ba1c_66af0464555e.slice/crio-8082dda37b4f6ab15d98ccb57e994caa7d9286c35453996c611a959e7d464b2a WatchSource:0}: Error finding container 8082dda37b4f6ab15d98ccb57e994caa7d9286c35453996c611a959e7d464b2a: Status 404 returned error can't find the container with id 8082dda37b4f6ab15d98ccb57e994caa7d9286c35453996c611a959e7d464b2a
Nov 03 21:15:02.134960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.134934    5533 generic.go:356] "Generic (PLEG): container finished" podID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerID="500a2533ca6975403675ee5ead4784f0221f47ea66d75060f78060e6a9959e1f" exitCode=60
Nov 03 21:15:02.135297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.135019    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerDied","Data":"500a2533ca6975403675ee5ead4784f0221f47ea66d75060f78060e6a9959e1f"}
Nov 03 21:15:02.135297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.135242    5533 scope.go:122] "RemoveContainer" containerID="500a2533ca6975403675ee5ead4784f0221f47ea66d75060f78060e6a9959e1f"
Nov 03 21:15:02.136774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.136719    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerID="e0e8f11fd6fb682855b1f63cc7c5734bbeeb09613434b0e7692292a5065ebead" exitCode=60
Nov 03 21:15:02.136853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.136815    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" event={"ID":"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063","Type":"ContainerDied","Data":"e0e8f11fd6fb682855b1f63cc7c5734bbeeb09613434b0e7692292a5065ebead"}
Nov 03 21:15:02.137032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.137016    5533 scope.go:122] "RemoveContainer" containerID="e0e8f11fd6fb682855b1f63cc7c5734bbeeb09613434b0e7692292a5065ebead"
Nov 03 21:15:02.138567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.138512    5533 generic.go:356] "Generic (PLEG): container finished" podID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerID="78c6ee8f2df2739d6fe39ab606faa255838a7bfdbeff0a7c40743b52317f5440" exitCode=60
Nov 03 21:15:02.138731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.138576    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerDied","Data":"78c6ee8f2df2739d6fe39ab606faa255838a7bfdbeff0a7c40743b52317f5440"}
Nov 03 21:15:02.138822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.138793    5533 scope.go:122] "RemoveContainer" containerID="78c6ee8f2df2739d6fe39ab606faa255838a7bfdbeff0a7c40743b52317f5440"
Nov 03 21:15:02.140384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.140363    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerStarted","Data":"90ce7021d2e46e7d2822d791b7afe7809e45adbfb4a37c23cf2e8ef72c686617"}
Nov 03 21:15:02.140464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.140405    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerStarted","Data":"8082dda37b4f6ab15d98ccb57e994caa7d9286c35453996c611a959e7d464b2a"}
Nov 03 21:15:02.141956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.141937    5533 generic.go:356] "Generic (PLEG): container finished" podID="93555532-d28d-4b30-96a7-d217e8474cc5" containerID="4c0b0806fdb6706731b2c99bdb70f17444f425d7a555d6cd7b15fcfd77c1f55a" exitCode=60
Nov 03 21:15:02.142039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.142022    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerDied","Data":"4c0b0806fdb6706731b2c99bdb70f17444f425d7a555d6cd7b15fcfd77c1f55a"}
Nov 03 21:15:02.142214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.142196    5533 scope.go:122] "RemoveContainer" containerID="4c0b0806fdb6706731b2c99bdb70f17444f425d7a555d6cd7b15fcfd77c1f55a"
Nov 03 21:15:02.143691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.143668    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerStarted","Data":"6d541cb7c16f1cb993d568c7aaa3fb897c17a3dfd0d9baaf33789c697f94c516"}
Nov 03 21:15:02.143777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.143719    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerStarted","Data":"c4da6daee2db89e8ed66b90184185f46baadc39af3df058ccf6fdb95be60dd6b"}
Nov 03 21:15:02.145102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.145083    5533 generic.go:356] "Generic (PLEG): container finished" podID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerID="3ea46a3661fa96177a4d44e8c6b334a27ed7583abf892ae94d30ca3a26206ceb" exitCode=60
Nov 03 21:15:02.145179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.145109    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerDied","Data":"3ea46a3661fa96177a4d44e8c6b334a27ed7583abf892ae94d30ca3a26206ceb"}
Nov 03 21:15:02.145391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.145376    5533 scope.go:122] "RemoveContainer" containerID="3ea46a3661fa96177a4d44e8c6b334a27ed7583abf892ae94d30ca3a26206ceb"
Nov 03 21:15:02.499883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.499851    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eea2f5e-18f9-4814-ae68-1293dd23070d" path="/var/lib/kubelet/pods/8eea2f5e-18f9-4814-ae68-1293dd23070d/volumes"
Nov 03 21:15:02.500286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.500268    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:15:02.504989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.504965    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.549843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.549816    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:15:02.553889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.553866    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812243\""
Nov 03 21:15:02.554291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.553986    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812243-dockercfg-lbdzt\""
Nov 03 21:15:02.603745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.584080    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwk26\" (UniqueName: \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.603745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.584145    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.654015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.645829    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" podStartSLOduration=2.6458093849999997 podStartE2EDuration="2.645809385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:02.612046744 +0000 UTC m=+1604.243360222" watchObservedRunningTime="2026-11-03 21:15:02.645809385 +0000 UTC m=+1604.277122863"
Nov 03 21:15:02.724072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.724009    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwk26\" (UniqueName: \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.724336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.724314    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.726143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.725830    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.782717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.782584    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwk26\" (UniqueName: \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\") pod \"payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:02.997852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:02.997747    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" podStartSLOduration=2.99772947 podStartE2EDuration="2.99772947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:02.997145907 +0000 UTC m=+1604.628459374" watchObservedRunningTime="2026-11-03 21:15:02.99772947 +0000 UTC m=+1604.629042947"
Nov 03 21:15:03.028613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.028584    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:03.159226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.159166    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerStarted","Data":"21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"}
Nov 03 21:15:03.161979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.161926    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" event={"ID":"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063","Type":"ContainerStarted","Data":"dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183"}
Nov 03 21:15:03.164099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.164073    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerStarted","Data":"c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"}
Nov 03 21:15:03.166247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.166223    5533 generic.go:356] "Generic (PLEG): container finished" podID="dd885b08-7c24-491a-ba1c-66af0464555e" containerID="90ce7021d2e46e7d2822d791b7afe7809e45adbfb4a37c23cf2e8ef72c686617" exitCode=60
Nov 03 21:15:03.166488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.166310    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerDied","Data":"90ce7021d2e46e7d2822d791b7afe7809e45adbfb4a37c23cf2e8ef72c686617"}
Nov 03 21:15:03.166882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.166489    5533 scope.go:122] "RemoveContainer" containerID="90ce7021d2e46e7d2822d791b7afe7809e45adbfb4a37c23cf2e8ef72c686617"
Nov 03 21:15:03.168866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.168797    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerStarted","Data":"b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"}
Nov 03 21:15:03.173395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.172728    5533 generic.go:356] "Generic (PLEG): container finished" podID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerID="6d541cb7c16f1cb993d568c7aaa3fb897c17a3dfd0d9baaf33789c697f94c516" exitCode=60
Nov 03 21:15:03.173395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.172902    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerDied","Data":"6d541cb7c16f1cb993d568c7aaa3fb897c17a3dfd0d9baaf33789c697f94c516"}
Nov 03 21:15:03.173395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.173200    5533 scope.go:122] "RemoveContainer" containerID="6d541cb7c16f1cb993d568c7aaa3fb897c17a3dfd0d9baaf33789c697f94c516"
Nov 03 21:15:03.180100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.180077    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerStarted","Data":"49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"}
Nov 03 21:15:03.268910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.268826    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" podStartSLOduration=3.268814103 podStartE2EDuration="3.268814103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.267383997 +0000 UTC m=+1604.898697474" watchObservedRunningTime="2026-11-03 21:15:03.268814103 +0000 UTC m=+1604.900127580"
Nov 03 21:15:03.404625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.404595    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:15:03.489590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.489537    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" podStartSLOduration=3.489524412 podStartE2EDuration="3.489524412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.486530348 +0000 UTC m=+1605.117843828" watchObservedRunningTime="2026-11-03 21:15:03.489524412 +0000 UTC m=+1605.120837889"
Nov 03 21:15:03.599830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.599775    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" podStartSLOduration=3.599758778 podStartE2EDuration="3.599758778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.594539286 +0000 UTC m=+1605.225852762" watchObservedRunningTime="2026-11-03 21:15:03.599758778 +0000 UTC m=+1605.231072258"
Nov 03 21:15:03.814846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.814804    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" podStartSLOduration=3.814790227 podStartE2EDuration="3.814790227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.81144028 +0000 UTC m=+1605.442753771" watchObservedRunningTime="2026-11-03 21:15:03.814790227 +0000 UTC m=+1605.446103703"
Nov 03 21:15:03.921108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:03.921059    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" podStartSLOduration=3.921047096 podStartE2EDuration="3.921047096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.919767456 +0000 UTC m=+1605.551080945" watchObservedRunningTime="2026-11-03 21:15:03.921047096 +0000 UTC m=+1605.552360572"
Nov 03 21:15:04.191356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.191268    5533 generic.go:356] "Generic (PLEG): container finished" podID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerID="49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6" exitCode=60
Nov 03 21:15:04.191806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.191360    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerDied","Data":"49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"}
Nov 03 21:15:04.191806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.191432    5533 scope.go:122] "RemoveContainer" containerID="3ea46a3661fa96177a4d44e8c6b334a27ed7583abf892ae94d30ca3a26206ceb"
Nov 03 21:15:04.191806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.191677    5533 scope.go:122] "RemoveContainer" containerID="49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"
Nov 03 21:15:04.192046 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:04.192017    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5_odh-ai-gateway-infra(39fd72d9-c37b-45ec-a377-3dab0d6e029a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a"
Nov 03 21:15:04.194037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.194014    5533 generic.go:356] "Generic (PLEG): container finished" podID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerID="21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae" exitCode=60
Nov 03 21:15:04.194155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.194124    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerDied","Data":"21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"}
Nov 03 21:15:04.194354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.194337    5533 scope.go:122] "RemoveContainer" containerID="21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"
Nov 03 21:15:04.194707 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:04.194674    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-4mq8z_odh-ai-gateway-infra(efe1c532-b595-4732-8a51-5a82da69f5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0"
Nov 03 21:15:04.196667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.196615    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerID="dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183" exitCode=60
Nov 03 21:15:04.196779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.196744    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" event={"ID":"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063","Type":"ContainerDied","Data":"dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183"}
Nov 03 21:15:04.197131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.196924    5533 scope.go:122] "RemoveContainer" containerID="dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183"
Nov 03 21:15:04.197214 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:04.197199    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg_odh-ai-gateway-infra(6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063"
Nov 03 21:15:04.198986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.198964    5533 generic.go:356] "Generic (PLEG): container finished" podID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerID="c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd" exitCode=60
Nov 03 21:15:04.199088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.199029    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerDied","Data":"c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"}
Nov 03 21:15:04.199332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.199316    5533 scope.go:122] "RemoveContainer" containerID="c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"
Nov 03 21:15:04.199590 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:04.199566    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp_odh-ai-gateway-infra(2e536b92-58e4-420c-bbaf-e23ca396a048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048"
Nov 03 21:15:04.201878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.201853    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerStarted","Data":"da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"}
Nov 03 21:15:04.203887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.203867    5533 generic.go:356] "Generic (PLEG): container finished" podID="93555532-d28d-4b30-96a7-d217e8474cc5" containerID="b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f" exitCode=60
Nov 03 21:15:04.203986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.203956    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerDied","Data":"b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"}
Nov 03 21:15:04.204155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.204132    5533 scope.go:122] "RemoveContainer" containerID="b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"
Nov 03 21:15:04.204376 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:04.204357    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f_odh-ai-gateway-infra(93555532-d28d-4b30-96a7-d217e8474cc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" podUID="93555532-d28d-4b30-96a7-d217e8474cc5"
Nov 03 21:15:04.205901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.205878    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerStarted","Data":"1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"}
Nov 03 21:15:04.207591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.207572    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" event={"ID":"8e3f7bb6-a586-480c-9fba-456531e18224","Type":"ContainerStarted","Data":"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"}
Nov 03 21:15:04.207768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.207605    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" event={"ID":"8e3f7bb6-a586-480c-9fba-456531e18224","Type":"ContainerStarted","Data":"775bbba79fefc1569ded9d86cdd0aa7b211fcaeade71785e5d76a84d08ade5cb"}
Nov 03 21:15:04.207768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.207651    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:04.208235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.208208    5533 scope.go:122] "RemoveContainer" containerID="500a2533ca6975403675ee5ead4784f0221f47ea66d75060f78060e6a9959e1f"
Nov 03 21:15:04.221263 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.221246    5533 scope.go:122] "RemoveContainer" containerID="e0e8f11fd6fb682855b1f63cc7c5734bbeeb09613434b0e7692292a5065ebead"
Nov 03 21:15:04.233701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.233685    5533 scope.go:122] "RemoveContainer" containerID="78c6ee8f2df2739d6fe39ab606faa255838a7bfdbeff0a7c40743b52317f5440"
Nov 03 21:15:04.255464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.255424    5533 scope.go:122] "RemoveContainer" containerID="4c0b0806fdb6706731b2c99bdb70f17444f425d7a555d6cd7b15fcfd77c1f55a"
Nov 03 21:15:04.666982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.666893    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:04.959945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:04.959879    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" podStartSLOduration=2.959860695 podStartE2EDuration="2.959860695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:04.957150888 +0000 UTC m=+1606.588464365" watchObservedRunningTime="2026-11-03 21:15:04.959860695 +0000 UTC m=+1606.591174172"
Nov 03 21:15:05.220352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.220271    5533 scope.go:122] "RemoveContainer" containerID="49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"
Nov 03 21:15:05.220762 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.220620    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5_odh-ai-gateway-infra(39fd72d9-c37b-45ec-a377-3dab0d6e029a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a"
Nov 03 21:15:05.222129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.222107    5533 scope.go:122] "RemoveContainer" containerID="21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"
Nov 03 21:15:05.222441 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.222421    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-4mq8z_odh-ai-gateway-infra(efe1c532-b595-4732-8a51-5a82da69f5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0"
Nov 03 21:15:05.223784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.223757    5533 scope.go:122] "RemoveContainer" containerID="dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183"
Nov 03 21:15:05.224100 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.224073    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg_odh-ai-gateway-infra(6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063"
Nov 03 21:15:05.225540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.225521    5533 scope.go:122] "RemoveContainer" containerID="c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"
Nov 03 21:15:05.225835 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.225811    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp_odh-ai-gateway-infra(2e536b92-58e4-420c-bbaf-e23ca396a048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048"
Nov 03 21:15:05.227117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.227094    5533 generic.go:356] "Generic (PLEG): container finished" podID="dd885b08-7c24-491a-ba1c-66af0464555e" containerID="da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a" exitCode=60
Nov 03 21:15:05.227205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.227179    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerDied","Data":"da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"}
Nov 03 21:15:05.227255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.227230    5533 scope.go:122] "RemoveContainer" containerID="90ce7021d2e46e7d2822d791b7afe7809e45adbfb4a37c23cf2e8ef72c686617"
Nov 03 21:15:05.227367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.227350    5533 scope.go:122] "RemoveContainer" containerID="da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"
Nov 03 21:15:05.227587 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.227564    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv_odh-ai-gateway-infra(dd885b08-7c24-491a-ba1c-66af0464555e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" podUID="dd885b08-7c24-491a-ba1c-66af0464555e"
Nov 03 21:15:05.229094 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.229074    5533 scope.go:122] "RemoveContainer" containerID="b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"
Nov 03 21:15:05.229324 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.229306    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f_odh-ai-gateway-infra(93555532-d28d-4b30-96a7-d217e8474cc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" podUID="93555532-d28d-4b30-96a7-d217e8474cc5"
Nov 03 21:15:05.230598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.230581    5533 generic.go:356] "Generic (PLEG): container finished" podID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerID="1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904" exitCode=60
Nov 03 21:15:05.230682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.230666    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerDied","Data":"1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"}
Nov 03 21:15:05.230861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.230846    5533 scope.go:122] "RemoveContainer" containerID="1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"
Nov 03 21:15:05.231060 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:05.231044    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh_odh-ai-gateway-infra(0fb9bb06-b1f0-4042-8110-f4033cfb9205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205"
Nov 03 21:15:05.241233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:05.241214    5533 scope.go:122] "RemoveContainer" containerID="6d541cb7c16f1cb993d568c7aaa3fb897c17a3dfd0d9baaf33789c697f94c516"
Nov 03 21:15:06.241631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.241605    5533 scope.go:122] "RemoveContainer" containerID="da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"
Nov 03 21:15:06.242103 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:06.241892    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv_odh-ai-gateway-infra(dd885b08-7c24-491a-ba1c-66af0464555e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" podUID="dd885b08-7c24-491a-ba1c-66af0464555e"
Nov 03 21:15:06.243352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.243329    5533 scope.go:122] "RemoveContainer" containerID="1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"
Nov 03 21:15:06.243769 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:06.243553    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh_odh-ai-gateway-infra(0fb9bb06-b1f0-4042-8110-f4033cfb9205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205"
Nov 03 21:15:06.408464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.408439    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:06.485992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.485948    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\" (UniqueName: \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\") pod \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") "
Nov 03 21:15:06.486179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.486159    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\") pod \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\" (UID: \"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063\") "
Nov 03 21:15:06.486572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.486538    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca" pod "6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" (UID: "6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:06.488309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.488284    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz" pod "6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" (UID: "6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063"). InnerVolumeSpecName "kube-api-access-f4xhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:06.587663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.587607    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:06.587811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:06.587675    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xhz\" (UniqueName: \"kubernetes.io/projected/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063-kube-api-access-f4xhz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:07.258948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:07.258906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg" event={"ID":"6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063","Type":"ContainerDied","Data":"b2b2fb0484c6fd8ee57c13a719654754492436a23431cb8d8729c5057a55a9c6"}
Nov 03 21:15:07.259464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:07.258989    5533 scope.go:122] "RemoveContainer" containerID="dc53c9f84d39ffc61f1e10075683a8cb23059fc23d605a5d1a2e07dda703d183"
Nov 03 21:15:07.259464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:07.259057    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"
Nov 03 21:15:07.415146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:07.415111    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:07.462476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:07.462443    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3337df-29895675-t88zg"]
Nov 03 21:15:08.416782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:08.416731    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:15:08.495814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:08.495783    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" path="/var/lib/kubelet/pods/6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063/volumes"
Nov 03 21:15:14.503074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:14.503046    5533 scope.go:122] "RemoveContainer" containerID="c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"
Nov 03 21:15:14.503940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:14.503919    5533 scope.go:122] "RemoveContainer" containerID="21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"
Nov 03 21:15:14.504522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:14.504502    5533 scope.go:122] "RemoveContainer" containerID="b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"
Nov 03 21:15:14.511338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:14.511311    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:15:14.511499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:14.511476    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy" containerID="cri-o://25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0" gracePeriod=30
Nov 03 21:15:15.234115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:15.234084    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:15:15.334284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:15.334245    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerStarted","Data":"85ac70eb23c77af9245142bfb919db4cfc39be661d1bfcc664ff3261958a74d4"}
Nov 03 21:15:15.336276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:15.336243    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerStarted","Data":"977e74f3ed4a5024c2dc0fcacf7adf8b925b3f13026d6d459497443cd9f1a318"}
Nov 03 21:15:15.338073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:15.338052    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerStarted","Data":"bfc88a92d3b74cd701b7a3333e8614317f9be0a09ad62c6c3938251a7b861224"}
Nov 03 21:15:16.350013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.349982    5533 generic.go:356] "Generic (PLEG): container finished" podID="93555532-d28d-4b30-96a7-d217e8474cc5" containerID="bfc88a92d3b74cd701b7a3333e8614317f9be0a09ad62c6c3938251a7b861224" exitCode=60
Nov 03 21:15:16.350498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.350035    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerDied","Data":"bfc88a92d3b74cd701b7a3333e8614317f9be0a09ad62c6c3938251a7b861224"}
Nov 03 21:15:16.350498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.350109    5533 scope.go:122] "RemoveContainer" containerID="b7063774df3f48700cfa79a40ebc09a5f444d8065c2889d42b166d4f7900e08f"
Nov 03 21:15:16.350621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.350505    5533 scope.go:122] "RemoveContainer" containerID="bfc88a92d3b74cd701b7a3333e8614317f9be0a09ad62c6c3938251a7b861224"
Nov 03 21:15:16.350911 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:16.350882    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f_odh-ai-gateway-infra(93555532-d28d-4b30-96a7-d217e8474cc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" podUID="93555532-d28d-4b30-96a7-d217e8474cc5"
Nov 03 21:15:16.352979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.352947    5533 generic.go:356] "Generic (PLEG): container finished" podID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerID="85ac70eb23c77af9245142bfb919db4cfc39be661d1bfcc664ff3261958a74d4" exitCode=60
Nov 03 21:15:16.353087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.353005    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerDied","Data":"85ac70eb23c77af9245142bfb919db4cfc39be661d1bfcc664ff3261958a74d4"}
Nov 03 21:15:16.353426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.353409    5533 scope.go:122] "RemoveContainer" containerID="85ac70eb23c77af9245142bfb919db4cfc39be661d1bfcc664ff3261958a74d4"
Nov 03 21:15:16.353803 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:16.353774    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-4mq8z_odh-ai-gateway-infra(efe1c532-b595-4732-8a51-5a82da69f5d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0"
Nov 03 21:15:16.355450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.355428    5533 generic.go:356] "Generic (PLEG): container finished" podID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerID="977e74f3ed4a5024c2dc0fcacf7adf8b925b3f13026d6d459497443cd9f1a318" exitCode=60
Nov 03 21:15:16.355584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.355506    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerDied","Data":"977e74f3ed4a5024c2dc0fcacf7adf8b925b3f13026d6d459497443cd9f1a318"}
Nov 03 21:15:16.355936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.355917    5533 scope.go:122] "RemoveContainer" containerID="977e74f3ed4a5024c2dc0fcacf7adf8b925b3f13026d6d459497443cd9f1a318"
Nov 03 21:15:16.356296 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:16.356213    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp_odh-ai-gateway-infra(2e536b92-58e4-420c-bbaf-e23ca396a048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048"
Nov 03 21:15:16.366439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.366359    5533 scope.go:122] "RemoveContainer" containerID="21c2928c06c1cdccb65460637b716a77af4156ddb463810cc81894db22927aae"
Nov 03 21:15:16.381440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.381419    5533 scope.go:122] "RemoveContainer" containerID="c2f46408a2f69ff79996a95a5eb6ccbb74c41be5dbfdf9563f377e55007b5fbd"
Nov 03 21:15:16.474873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.474826    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:16.500890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.500612    5533 scope.go:122] "RemoveContainer" containerID="49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"
Nov 03 21:15:16.576195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.576160    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:16.587994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.587948    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:15:16.588926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.588891    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.588926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.588917    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.589205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.588943    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.589205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.588952    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.589205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.589141    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.589205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.589164    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d01da5c-b90d-4c0f-8ecb-6d2ec02e1063" containerName="cleanup"
Nov 03 21:15:16.594619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.594587    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.678660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.658740    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-66513c-openshift-default-dockercfg-q6wpb\""
Nov 03 21:15:16.703664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.703414    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.703664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.703498    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.703664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.703589    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.703664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.703617    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.704083    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.704171    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84hv\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.704286    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.704339    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.704408    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.720132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.714694    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:15:16.787154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.778311    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:16.809065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809020    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809083    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809118    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809177    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84hv\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809357    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809422    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809495    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809529    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.809778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.809591    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.810154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.810108    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.810495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.810385    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.811800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.811364    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.811800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.811742    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.812596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.812551    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.815262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.815234    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.816873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.816830    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.876976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.876904    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84hv\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.886664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.886598    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\") pod \"e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") " pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:16.912646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:16.912598    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:17.369051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.369012    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerStarted","Data":"6e167c468fb01c6b75b3ed604d880e6d90b472d9cd301a3355eddef3e275ea4b"}
Nov 03 21:15:17.380385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.380348    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:15:17.383077 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:17.383014    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00acc07_571e_43cc_820d_1768a228c626.slice/crio-f087e9a463386318a09ae9b478e1a26779f359ec2288dc415adeaa20d4dd846f WatchSource:0}: Error finding container f087e9a463386318a09ae9b478e1a26779f359ec2288dc415adeaa20d4dd846f: Status 404 returned error can't find the container with id f087e9a463386318a09ae9b478e1a26779f359ec2288dc415adeaa20d4dd846f
Nov 03 21:15:17.387490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.387453    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:17.387607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.387535    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:17.387607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.387575    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:17.487521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.486273    5533 scope.go:122] "RemoveContainer" containerID="da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"
Nov 03 21:15:17.487521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.487150    5533 scope.go:122] "RemoveContainer" containerID="1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"
Nov 03 21:15:17.723333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.723247    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:17.727325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.727302    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:17.748234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.748210    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:17.845424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845395    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\") pod \"2e536b92-58e4-420c-bbaf-e23ca396a048\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") "
Nov 03 21:15:17.845568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845466    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\" (UniqueName: \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\") pod \"efe1c532-b595-4732-8a51-5a82da69f5d0\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") "
Nov 03 21:15:17.845568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845488    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\") pod \"efe1c532-b595-4732-8a51-5a82da69f5d0\" (UID: \"efe1c532-b595-4732-8a51-5a82da69f5d0\") "
Nov 03 21:15:17.845710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845612    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\") pod \"2e536b92-58e4-420c-bbaf-e23ca396a048\" (UID: \"2e536b92-58e4-420c-bbaf-e23ca396a048\") "
Nov 03 21:15:17.845803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845786    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\") pod \"93555532-d28d-4b30-96a7-d217e8474cc5\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") "
Nov 03 21:15:17.845877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845859    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca" pod "efe1c532-b595-4732-8a51-5a82da69f5d0" (UID: "efe1c532-b595-4732-8a51-5a82da69f5d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.846174 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.845885    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\" (UniqueName: \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\") pod \"93555532-d28d-4b30-96a7-d217e8474cc5\" (UID: \"93555532-d28d-4b30-96a7-d217e8474cc5\") "
Nov 03 21:15:17.846244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.846184    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca" pod "93555532-d28d-4b30-96a7-d217e8474cc5" (UID: "93555532-d28d-4b30-96a7-d217e8474cc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.846784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.846763    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe1c532-b595-4732-8a51-5a82da69f5d0-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.846940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.846916    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93555532-d28d-4b30-96a7-d217e8474cc5-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.846940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.846878    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca" pod "2e536b92-58e4-420c-bbaf-e23ca396a048" (UID: "2e536b92-58e4-420c-bbaf-e23ca396a048"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.854894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.854873    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b" pod "2e536b92-58e4-420c-bbaf-e23ca396a048" (UID: "2e536b92-58e4-420c-bbaf-e23ca396a048"). InnerVolumeSpecName "kube-api-access-2zv7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.855129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.855099    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx" pod "93555532-d28d-4b30-96a7-d217e8474cc5" (UID: "93555532-d28d-4b30-96a7-d217e8474cc5"). InnerVolumeSpecName "kube-api-access-dnwjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.856091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.856059    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh" pod "efe1c532-b595-4732-8a51-5a82da69f5d0" (UID: "efe1c532-b595-4732-8a51-5a82da69f5d0"). InnerVolumeSpecName "kube-api-access-4fnmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.947735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.947696    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e536b92-58e4-420c-bbaf-e23ca396a048-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.947735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.947734    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnmh\" (UniqueName: \"kubernetes.io/projected/efe1c532-b595-4732-8a51-5a82da69f5d0-kube-api-access-4fnmh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.947921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.947750    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/2e536b92-58e4-420c-bbaf-e23ca396a048-kube-api-access-2zv7b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.947921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:17.947760    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnwjx\" (UniqueName: \"kubernetes.io/projected/93555532-d28d-4b30-96a7-d217e8474cc5-kube-api-access-dnwjx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.391545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.391513    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"
Nov 03 21:15:18.392069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.391512    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp" event={"ID":"2e536b92-58e4-420c-bbaf-e23ca396a048","Type":"ContainerDied","Data":"0b5eec2a118a861bffc66a3b2db400eae65b451d7fd567dc174e06f334d427fa"}
Nov 03 21:15:18.392069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.391681    5533 scope.go:122] "RemoveContainer" containerID="977e74f3ed4a5024c2dc0fcacf7adf8b925b3f13026d6d459497443cd9f1a318"
Nov 03 21:15:18.394388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.394362    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerStarted","Data":"bbfe16aba31482c22c563c885e3eebe2ecb5434a376467d2bff1873123ccad88"}
Nov 03 21:15:18.396222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.396196    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"
Nov 03 21:15:18.396340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.396221    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f" event={"ID":"93555532-d28d-4b30-96a7-d217e8474cc5","Type":"ContainerDied","Data":"3b2b47e9557647003f9b3b4c381673cd5af4d7d3233467e94935d24e9cecc28a"}
Nov 03 21:15:18.398346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.398322    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerStarted","Data":"af5ec4790bed71fbb07d2f9177f51dd055e41e1b95174058c78404c61f7c97ed"}
Nov 03 21:15:18.401975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.401935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerStarted","Data":"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"}
Nov 03 21:15:18.402095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.402033    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerStarted","Data":"f087e9a463386318a09ae9b478e1a26779f359ec2288dc415adeaa20d4dd846f"}
Nov 03 21:15:18.404326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.404305    5533 generic.go:356] "Generic (PLEG): container finished" podID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerID="6e167c468fb01c6b75b3ed604d880e6d90b472d9cd301a3355eddef3e275ea4b" exitCode=60
Nov 03 21:15:18.404410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.404383    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerDied","Data":"6e167c468fb01c6b75b3ed604d880e6d90b472d9cd301a3355eddef3e275ea4b"}
Nov 03 21:15:18.404734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.404718    5533 scope.go:122] "RemoveContainer" containerID="6e167c468fb01c6b75b3ed604d880e6d90b472d9cd301a3355eddef3e275ea4b"
Nov 03 21:15:18.404996 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:18.404972    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5_odh-ai-gateway-infra(39fd72d9-c37b-45ec-a377-3dab0d6e029a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a"
Nov 03 21:15:18.405794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.405776    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z" event={"ID":"efe1c532-b595-4732-8a51-5a82da69f5d0","Type":"ContainerDied","Data":"dcec2133e1250896c2be0922f638a270b5b39fcb7260d4d3c1b7001797890cc1"}
Nov 03 21:15:18.405794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.405793    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"
Nov 03 21:15:18.506432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.506368    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:18.511048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.511023    5533 scope.go:122] "RemoveContainer" containerID="bfc88a92d3b74cd701b7a3333e8614317f9be0a09ad62c6c3938251a7b861224"
Nov 03 21:15:18.522768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.522747    5533 scope.go:122] "RemoveContainer" containerID="49c9bd208d77de3eaef53602520d94cb33ca6519cb0537924177b4494ac0f8a6"
Nov 03 21:15:18.538462 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.538444    5533 scope.go:122] "RemoveContainer" containerID="85ac70eb23c77af9245142bfb919db4cfc39be661d1bfcc664ff3261958a74d4"
Nov 03 21:15:18.662697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.662078    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:18.664256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.664232    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895675-8xtgp"]
Nov 03 21:15:18.774531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.774482    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podStartSLOduration=2.7744679960000003 podStartE2EDuration="2.774467996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:18.77088615 +0000 UTC m=+1620.402199641" watchObservedRunningTime="2026-11-03 21:15:18.774467996 +0000 UTC m=+1620.405781684"
Nov 03 21:15:18.915767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.914490    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:18.921924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:18.921894    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:19.098151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.098118    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:19.102494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.102465    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-22600c-29895675-7ck8f"]
Nov 03 21:15:19.105946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.105924    5533 scope.go:122] "RemoveContainer" containerID="d8ac4cb42f3ce6cd0f37f5ede1f1bc5b0a2061a8a1cb15f0ffad50bd2788f4ce"
Nov 03 21:15:19.123899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.123838    5533 scope.go:122] "RemoveContainer" containerID="7619c1480f2bcde3db6bd16e185c205c3dac0abd8d6fedc3f806bbab23e5af55"
Nov 03 21:15:19.133815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.133794    5533 scope.go:122] "RemoveContainer" containerID="4f1782955156057a87a68bb1352c68d445175d3d538223e41de40919ad317f91"
Nov 03 21:15:19.142631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.142608    5533 scope.go:122] "RemoveContainer" containerID="635a534ed255204f7cf57e80d2d2946e12b9a7dd37d056958bba55cd494bc91a"
Nov 03 21:15:19.151482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.151463    5533 scope.go:122] "RemoveContainer" containerID="8dc15af8b544dad2ba8f32adc0a40bdc37afc8d38ec0cd12f46afbb79b33f20e"
Nov 03 21:15:19.161486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.161468    5533 scope.go:122] "RemoveContainer" containerID="b4bd181e57694416ac4285e692b5153d67d6a4887bffd072a0cdece938a36993"
Nov 03 21:15:19.171533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.171518    5533 scope.go:122] "RemoveContainer" containerID="a557d0170b467b8efdc35f8083b4a5a09a64627805fbafcfa9daafe4e56edad4"
Nov 03 21:15:19.179795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.179776    5533 scope.go:122] "RemoveContainer" containerID="2dbd0fc354054bc08392a6afe00a17dba799f8aa534d1430626b2ba127c60a98"
Nov 03 21:15:19.188054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.188022    5533 scope.go:122] "RemoveContainer" containerID="633c823fe9a344d76a8e7f94e89ad1a0161c8521124d524b805a5584e5be9fb0"
Nov 03 21:15:19.196892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.196860    5533 scope.go:122] "RemoveContainer" containerID="ecc072b07eae4899cb7d426708e5484ca1de0ce5633902c6a3f115568f14e16a"
Nov 03 21:15:19.205795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.205772    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:19.215162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.215117    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-4mq8z"]
Nov 03 21:15:19.270203 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:19.270170    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:15:19.413461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.413437    5533 generic.go:356] "Generic (PLEG): container finished" podID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerID="f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89" exitCode=0
Nov 03 21:15:19.413817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.413511    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerDied","Data":"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"}
Nov 03 21:15:19.414018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.413992    5533 scope.go:122] "RemoveContainer" containerID="f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"
Nov 03 21:15:19.415510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.415484    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:19.415579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.415550    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:19.415620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.415582    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:15:19.422457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.422375    5533 generic.go:356] "Generic (PLEG): container finished" podID="dd885b08-7c24-491a-ba1c-66af0464555e" containerID="bbfe16aba31482c22c563c885e3eebe2ecb5434a376467d2bff1873123ccad88" exitCode=60
Nov 03 21:15:19.422457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.422398    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerDied","Data":"bbfe16aba31482c22c563c885e3eebe2ecb5434a376467d2bff1873123ccad88"}
Nov 03 21:15:19.422675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.422472    5533 scope.go:122] "RemoveContainer" containerID="da8c60002709d86a920e747730d14bae83e4567aa59ac38359c1cb4d5310ec8a"
Nov 03 21:15:19.422775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.422759    5533 scope.go:122] "RemoveContainer" containerID="bbfe16aba31482c22c563c885e3eebe2ecb5434a376467d2bff1873123ccad88"
Nov 03 21:15:19.423135 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:19.423032    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv_odh-ai-gateway-infra(dd885b08-7c24-491a-ba1c-66af0464555e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" podUID="dd885b08-7c24-491a-ba1c-66af0464555e"
Nov 03 21:15:19.424732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.424712    5533 generic.go:356] "Generic (PLEG): container finished" podID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerID="af5ec4790bed71fbb07d2f9177f51dd055e41e1b95174058c78404c61f7c97ed" exitCode=60
Nov 03 21:15:19.424822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.424740    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerDied","Data":"af5ec4790bed71fbb07d2f9177f51dd055e41e1b95174058c78404c61f7c97ed"}
Nov 03 21:15:19.425100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.425078    5533 scope.go:122] "RemoveContainer" containerID="af5ec4790bed71fbb07d2f9177f51dd055e41e1b95174058c78404c61f7c97ed"
Nov 03 21:15:19.425100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.425095    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:19.425396 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:19.425373    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh_odh-ai-gateway-infra(0fb9bb06-b1f0-4042-8110-f4033cfb9205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205"
Nov 03 21:15:19.426116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.426098    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:15:19.436497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.436482    5533 scope.go:122] "RemoveContainer" containerID="1772d0e8673d5f73ba82b1da9f1e9fedf9466d9af335203e5417727077263904"
Nov 03 21:15:19.574447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.574409    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:19.655236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.655212    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:19.777914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.777882    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\" (UniqueName: \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\") pod \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") "
Nov 03 21:15:19.778096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.778002    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\") pod \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\" (UID: \"39fd72d9-c37b-45ec-a377-3dab0d6e029a\") "
Nov 03 21:15:19.788674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.786752    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca" pod "39fd72d9-c37b-45ec-a377-3dab0d6e029a" (UID: "39fd72d9-c37b-45ec-a377-3dab0d6e029a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:19.788674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.786923    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6" pod "39fd72d9-c37b-45ec-a377-3dab0d6e029a" (UID: "39fd72d9-c37b-45ec-a377-3dab0d6e029a"). InnerVolumeSpecName "kube-api-access-rlfb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:19.808519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.808493    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:15:19.879527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.879496    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfb6\" (UniqueName: \"kubernetes.io/projected/39fd72d9-c37b-45ec-a377-3dab0d6e029a-kube-api-access-rlfb6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.879527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.879525    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fd72d9-c37b-45ec-a377-3dab0d6e029a-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.980550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980444    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.980550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980499    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.980808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980551    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.980808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980706    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.980808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980739    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.980808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980790    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.981020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980823    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.981020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980868    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.981020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.980924    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\") pod \"675ce686-5167-422d-89d8-a6a1c15ef3de\" (UID: \"675ce686-5167-422d-89d8-a6a1c15ef3de\") "
Nov 03 21:15:19.981306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:19.981306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981286    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:19.981885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981552    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:19.981885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981590    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.981885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981616    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.981885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.981838    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:19.982097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.982004    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:19.983559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.983328    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:19.984222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.984199    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:15:19.984859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.984834    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:19.985040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:19.985021    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq" pod "675ce686-5167-422d-89d8-a6a1c15ef3de" (UID: "675ce686-5167-422d-89d8-a6a1c15ef3de"). InnerVolumeSpecName "kube-api-access-kxqrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:20.002334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.002299    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.082916    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675ce686-5167-422d-89d8-a6a1c15ef3de-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.082956    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.082973    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.082988    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqrq\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-kube-api-access-kxqrq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.083004    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.083019    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.083069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.083032    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675ce686-5167-422d-89d8-a6a1c15ef3de-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.437176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.437145    5533 generic.go:356] "Generic (PLEG): container finished" podID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerID="25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0" exitCode=0
Nov 03 21:15:20.437578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.437303    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerDied","Data":"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"}
Nov 03 21:15:20.437578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.437354    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t" event={"ID":"675ce686-5167-422d-89d8-a6a1c15ef3de","Type":"ContainerDied","Data":"eca31102370cff03670d6fda1c91904780345495a92a4a09fb039763790f7392"}
Nov 03 21:15:20.437578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.437384    5533 scope.go:122] "RemoveContainer" containerID="25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"
Nov 03 21:15:20.437578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.437545    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"
Nov 03 21:15:20.441152 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.441120    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerStarted","Data":"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"}
Nov 03 21:15:20.444273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.444238    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"
Nov 03 21:15:20.444714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.444687    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5" event={"ID":"39fd72d9-c37b-45ec-a377-3dab0d6e029a","Type":"ContainerDied","Data":"01325354da81c8b86d202b9e12830830f817d7d4120f800b03bd1d7eaf7ac7e9"}
Nov 03 21:15:20.450500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.450467    5533 scope.go:122] "RemoveContainer" containerID="f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"
Nov 03 21:15:20.497790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.497549    5533 scope.go:122] "RemoveContainer" containerID="25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"
Nov 03 21:15:20.497926 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:20.497880    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0\": container with ID starting with 25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0 not found: ID does not exist" containerID="25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"
Nov 03 21:15:20.497926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.497906    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0"} err="failed to get container status \"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0\": rpc error: code = NotFound desc = could not find container \"25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0\": container with ID starting with 25eb88494b1620eadaef32163b7acf750229d08fc43a8507f0eed29fa63572c0 not found: ID does not exist"
Nov 03 21:15:20.497926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.497924    5533 scope.go:122] "RemoveContainer" containerID="f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"
Nov 03 21:15:20.498841 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:20.498259    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16\": container with ID starting with f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16 not found: ID does not exist" containerID="f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"
Nov 03 21:15:20.498841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.498299    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" path="/var/lib/kubelet/pods/2e536b92-58e4-420c-bbaf-e23ca396a048/volumes"
Nov 03 21:15:20.498841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.498686    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16"} err="failed to get container status \"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16\": rpc error: code = NotFound desc = could not find container \"f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16\": container with ID starting with f5c91c9baeb3ff62b66fb03d188770246052291e03fea0c605e73446bd796f16 not found: ID does not exist"
Nov 03 21:15:20.498841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.498744    5533 scope.go:122] "RemoveContainer" containerID="6e167c468fb01c6b75b3ed604d880e6d90b472d9cd301a3355eddef3e275ea4b"
Nov 03 21:15:20.499411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.499030    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" path="/var/lib/kubelet/pods/93555532-d28d-4b30-96a7-d217e8474cc5/volumes"
Nov 03 21:15:20.499560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.499539    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" path="/var/lib/kubelet/pods/efe1c532-b595-4732-8a51-5a82da69f5d0/volumes"
Nov 03 21:15:20.537104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.537081    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:15:20.537210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.537188    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:15:20.537288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.537228    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 21:15:20.692382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.692356    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:20.694358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.694331    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895675-bdvs5"]
Nov 03 21:15:20.699725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.699706    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:20.717476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.717454    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:20.802788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.802754    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\" (UniqueName: \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\") pod \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") "
Nov 03 21:15:20.802967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.802810    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\") pod \"dd885b08-7c24-491a-ba1c-66af0464555e\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") "
Nov 03 21:15:20.802967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.802956    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\") pod \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\" (UID: \"0fb9bb06-b1f0-4042-8110-f4033cfb9205\") "
Nov 03 21:15:20.803115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.803098    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\" (UniqueName: \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\") pod \"dd885b08-7c24-491a-ba1c-66af0464555e\" (UID: \"dd885b08-7c24-491a-ba1c-66af0464555e\") "
Nov 03 21:15:20.803192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.803138    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca" pod "dd885b08-7c24-491a-ba1c-66af0464555e" (UID: "dd885b08-7c24-491a-ba1c-66af0464555e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:20.803337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.803316    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca" pod "0fb9bb06-b1f0-4042-8110-f4033cfb9205" (UID: "0fb9bb06-b1f0-4042-8110-f4033cfb9205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:20.803691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.803671    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd885b08-7c24-491a-ba1c-66af0464555e-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.803769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.803694    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb9bb06-b1f0-4042-8110-f4033cfb9205-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.804859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.804835    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz" pod "0fb9bb06-b1f0-4042-8110-f4033cfb9205" (UID: "0fb9bb06-b1f0-4042-8110-f4033cfb9205"). InnerVolumeSpecName "kube-api-access-t9bxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:20.805068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.805048    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p" pod "dd885b08-7c24-491a-ba1c-66af0464555e" (UID: "dd885b08-7c24-491a-ba1c-66af0464555e"). InnerVolumeSpecName "kube-api-access-r2n8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:20.904687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.904647    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bxz\" (UniqueName: \"kubernetes.io/projected/0fb9bb06-b1f0-4042-8110-f4033cfb9205-kube-api-access-t9bxz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.904687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:20.904674    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2n8p\" (UniqueName: \"kubernetes.io/projected/dd885b08-7c24-491a-ba1c-66af0464555e-kube-api-access-r2n8p\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:21.016813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.016741    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:15:21.031958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.031932    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3337df-openshift-default-5bf8dcbd5f-n2f2t"]
Nov 03 21:15:21.484685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.484624    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"
Nov 03 21:15:21.484685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.484673    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv" event={"ID":"dd885b08-7c24-491a-ba1c-66af0464555e","Type":"ContainerDied","Data":"8082dda37b4f6ab15d98ccb57e994caa7d9286c35453996c611a959e7d464b2a"}
Nov 03 21:15:21.485215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.484762    5533 scope.go:122] "RemoveContainer" containerID="bbfe16aba31482c22c563c885e3eebe2ecb5434a376467d2bff1873123ccad88"
Nov 03 21:15:21.486047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.486001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh" event={"ID":"0fb9bb06-b1f0-4042-8110-f4033cfb9205","Type":"ContainerDied","Data":"c4da6daee2db89e8ed66b90184185f46baadc39af3df058ccf6fdb95be60dd6b"}
Nov 03 21:15:21.486047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.486037    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"
Nov 03 21:15:21.495544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.495526    5533 scope.go:122] "RemoveContainer" containerID="af5ec4790bed71fbb07d2f9177f51dd055e41e1b95174058c78404c61f7c97ed"
Nov 03 21:15:21.537227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.537199    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:15:21.537348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.537243    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 21:15:21.639768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.639737    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:21.643898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.643857    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0f8726-29895675-h47qh"]
Nov 03 21:15:21.745733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.745669    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:21.774652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:21.773729    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e92e3-29895675-fvxxv"]
Nov 03 21:15:22.494344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.494311    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" path="/var/lib/kubelet/pods/0fb9bb06-b1f0-4042-8110-f4033cfb9205/volumes"
Nov 03 21:15:22.494888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.494871    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" path="/var/lib/kubelet/pods/39fd72d9-c37b-45ec-a377-3dab0d6e029a/volumes"
Nov 03 21:15:22.495461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.495443    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" path="/var/lib/kubelet/pods/675ce686-5167-422d-89d8-a6a1c15ef3de/volumes"
Nov 03 21:15:22.496091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.496061    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" path="/var/lib/kubelet/pods/dd885b08-7c24-491a-ba1c-66af0464555e/volumes"
Nov 03 21:15:22.536972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.536946    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:15:22.537065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:22.537005    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 21:15:23.536697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:23.536659    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:15:23.537094 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:23.536734    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 21:15:24.537293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:24.537262    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 21:15:24.537687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:24.537311    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 21:15:25.580567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:25.580529    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:15:25.581442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:25.581421    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:15:25.582690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:25.582670    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:15:27.219309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:27.219266    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:15:31.680576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.680522    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683406    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683440    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683458    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683467    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683478    5533 cpu_manager.go:414] "containerMap: removing container" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683485    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683498    5533 cpu_manager.go:414] "containerMap: removing container" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683505    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683518    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683525    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683534    5533 cpu_manager.go:414] "containerMap: removing container" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683540    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683554    5533 cpu_manager.go:414] "containerMap: removing container" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683561    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683573    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683580    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683596    5533 cpu_manager.go:414] "containerMap: removing container" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683603    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683621    5533 cpu_manager.go:414] "containerMap: removing container" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683628    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683657    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683664    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683675    5533 cpu_manager.go:414] "containerMap: removing container" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683682    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683699    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683706    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683722    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683728    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683875    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683888    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683898    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683907    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683915    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683924    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683933    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683941    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683949    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683957    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683966    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683975    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683985    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:31.685188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.683994    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:31.710820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.693220    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.710820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.706984    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:15:31.743151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.743119    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66513c\""
Nov 03 21:15:31.789411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.789381    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66513c-dockercfg-tt2lc\""
Nov 03 21:15:31.794167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.794145    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.794351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.794330    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.900075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.900043    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.900198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.900182    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.900750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.900729    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:31.957910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:31.957856    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\") pod \"payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:32.019849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.019821    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:32.220644 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.220555    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" podUID="0dee5365-7f79-49fb-9742-140a7ed1acee" containerName="payload-pre-processing" containerID="cri-o://5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36" gracePeriod=25
Nov 03 21:15:32.391881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.391847    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:15:32.504656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.490541    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:15:32.608422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.608395    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\") pod \"0dee5365-7f79-49fb-9742-140a7ed1acee\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") "
Nov 03 21:15:32.608579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.608480    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\" (UniqueName: \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\") pod \"0dee5365-7f79-49fb-9742-140a7ed1acee\" (UID: \"0dee5365-7f79-49fb-9742-140a7ed1acee\") "
Nov 03 21:15:32.608838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.608809    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume" pod "0dee5365-7f79-49fb-9742-140a7ed1acee" (UID: "0dee5365-7f79-49fb-9742-140a7ed1acee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:32.608987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.608969    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dee5365-7f79-49fb-9742-140a7ed1acee-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:32.610783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.610756    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf" pod "0dee5365-7f79-49fb-9742-140a7ed1acee" (UID: "0dee5365-7f79-49fb-9742-140a7ed1acee"). InnerVolumeSpecName "kube-api-access-znfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:32.635414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.635384    5533 generic.go:356] "Generic (PLEG): container finished" podID="0dee5365-7f79-49fb-9742-140a7ed1acee" containerID="5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36" exitCode=0
Nov 03 21:15:32.635523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.635455    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"
Nov 03 21:15:32.635523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.635481    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" event={"ID":"0dee5365-7f79-49fb-9742-140a7ed1acee","Type":"ContainerDied","Data":"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"}
Nov 03 21:15:32.635610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.635554    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g" event={"ID":"0dee5365-7f79-49fb-9742-140a7ed1acee","Type":"ContainerDied","Data":"0d42649146b54932ce65c2684c407cde4582c19bbd301dde9944a476ceeb418e"}
Nov 03 21:15:32.635610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.635590    5533 scope.go:122] "RemoveContainer" containerID="5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"
Nov 03 21:15:32.637875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.637846    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" event={"ID":"9b9235ac-b0e7-4701-98a7-121e1beb1310","Type":"ContainerStarted","Data":"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"}
Nov 03 21:15:32.638002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.637897    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" event={"ID":"9b9235ac-b0e7-4701-98a7-121e1beb1310","Type":"ContainerStarted","Data":"24b304007bcaafcc4ee5ef65d667de2207df03ef1084c85a5150dca6df51f835"}
Nov 03 21:15:32.638002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.637944    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:32.647310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.647290    5533 scope.go:122] "RemoveContainer" containerID="5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"
Nov 03 21:15:32.647583 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:32.647562    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36\": container with ID starting with 5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36 not found: ID does not exist" containerID="5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"
Nov 03 21:15:32.647673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.647596    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36"} err="failed to get container status \"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36\": rpc error: code = NotFound desc = could not find container \"5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36\": container with ID starting with 5de9230a6fb815265c0a3d55110a51974f7253a29eddb3bb46e8e2738cebda36 not found: ID does not exist"
Nov 03 21:15:32.709806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.709785    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfvf\" (UniqueName: \"kubernetes.io/projected/0dee5365-7f79-49fb-9742-140a7ed1acee-kube-api-access-znfvf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:32.790894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.790833    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:15:32.796169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.796148    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e92e3-5fd5b97555-vjm5g"]
Nov 03 21:15:32.902848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:32.902796    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" podStartSLOduration=1.902784092 podStartE2EDuration="1.902784092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:32.899545695 +0000 UTC m=+1634.530859173" watchObservedRunningTime="2026-11-03 21:15:32.902784092 +0000 UTC m=+1634.534097569"
Nov 03 21:15:34.495525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:34.495486    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dee5365-7f79-49fb-9742-140a7ed1acee" path="/var/lib/kubelet/pods/0dee5365-7f79-49fb-9742-140a7ed1acee/volumes"
Nov 03 21:15:43.649065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:43.649037    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:15:46.548833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.548788    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549651    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549676    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549703    5533 cpu_manager.go:414] "containerMap: removing container" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549711    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675ce686-5167-422d-89d8-a6a1c15ef3de" containerName="istio-proxy"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549729    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0dee5365-7f79-49fb-9742-140a7ed1acee" containerName="payload-pre-processing"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549736    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dee5365-7f79-49fb-9742-140a7ed1acee" containerName="payload-pre-processing"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549753    5533 cpu_manager.go:414] "containerMap: removing container" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549760    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549776    5533 cpu_manager.go:414] "containerMap: removing container" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549784    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549837    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549845    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549858    5533 cpu_manager.go:414] "containerMap: removing container" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.549866    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550014    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e536b92-58e4-420c-bbaf-e23ca396a048" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550032    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb9bb06-b1f0-4042-8110-f4033cfb9205" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550042    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fd72d9-c37b-45ec-a377-3dab0d6e029a" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550052    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe1c532-b595-4732-8a51-5a82da69f5d0" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550063    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd885b08-7c24-491a-ba1c-66af0464555e" containerName="cleanup"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550076    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dee5365-7f79-49fb-9742-140a7ed1acee" containerName="payload-pre-processing"
Nov 03 21:15:46.554944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.550086    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="93555532-d28d-4b30-96a7-d217e8474cc5" containerName="cleanup"
Nov 03 21:15:46.576087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.576060    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.600684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.595021    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:15:46.605723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.605688    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctb6\" (UniqueName: \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.605845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.605817    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.634913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.634891    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9232e5-dockercfg-q62r2\""
Nov 03 21:15:46.686704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.686680    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9232e5\""
Nov 03 21:15:46.706511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.706485    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctb6\" (UniqueName: \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.706607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.706562    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.707228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.707211    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.769154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.769127    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctb6\" (UniqueName: \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\") pod \"payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:46.908217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:46.908158    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:47.279786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:47.279757    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:15:47.280547 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:15:47.280522    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9495d0d_e3cb_4dd0_bce7_1e70ce1bd8ec.slice/crio-96826dbbcb19077dc3b6743c6f77ea0421f955546736a2ca960eabccbf92eed4 WatchSource:0}: Error finding container 96826dbbcb19077dc3b6743c6f77ea0421f955546736a2ca960eabccbf92eed4: Status 404 returned error can't find the container with id 96826dbbcb19077dc3b6743c6f77ea0421f955546736a2ca960eabccbf92eed4
Nov 03 21:15:47.762427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:47.762392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" event={"ID":"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec","Type":"ContainerStarted","Data":"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"}
Nov 03 21:15:47.762936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:47.762447    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" event={"ID":"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec","Type":"ContainerStarted","Data":"96826dbbcb19077dc3b6743c6f77ea0421f955546736a2ca960eabccbf92eed4"}
Nov 03 21:15:47.762936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:47.762483    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:15:47.873966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:47.873912    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" podStartSLOduration=1.8738986880000001 podStartE2EDuration="1.873898688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:47.871253584 +0000 UTC m=+1649.502567062" watchObservedRunningTime="2026-11-03 21:15:47.873898688 +0000 UTC m=+1649.505212165"
Nov 03 21:15:49.261088 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:15:49.261054    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:15:56.827265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:56.827231    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:15:58.613111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:58.613074    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:15:58.777490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:15:58.777461    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:16:03.614466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.614374    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" podUID="f1cc166d-4a1a-4be6-bd24-0628b404b697" containerName="payload-pre-processing" containerID="cri-o://6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055" gracePeriod=25
Nov 03 21:16:03.888151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.888086    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:16:03.890304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.890280    5533 generic.go:356] "Generic (PLEG): container finished" podID="f1cc166d-4a1a-4be6-bd24-0628b404b697" containerID="6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055" exitCode=0
Nov 03 21:16:03.890458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.890320    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"
Nov 03 21:16:03.890458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.890331    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" event={"ID":"f1cc166d-4a1a-4be6-bd24-0628b404b697","Type":"ContainerDied","Data":"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"}
Nov 03 21:16:03.890458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.890370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw" event={"ID":"f1cc166d-4a1a-4be6-bd24-0628b404b697","Type":"ContainerDied","Data":"07e7946de5f2978bace054bc7c2de830c03bb402277e3b67a0035c8b2f970b35"}
Nov 03 21:16:03.890458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.890392    5533 scope.go:122] "RemoveContainer" containerID="6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"
Nov 03 21:16:03.901603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.901583    5533 scope.go:122] "RemoveContainer" containerID="6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"
Nov 03 21:16:03.901900 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:16:03.901868    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055\": container with ID starting with 6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055 not found: ID does not exist" containerID="6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"
Nov 03 21:16:03.901975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:03.901903    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055"} err="failed to get container status \"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055\": rpc error: code = NotFound desc = could not find container \"6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055\": container with ID starting with 6d7e7b8c210b083f5034d2ca832fcf04493093399ef7aaad52e3cb831e01b055 not found: ID does not exist"
Nov 03 21:16:04.020456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.020420    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\" (UniqueName: \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\") pod \"f1cc166d-4a1a-4be6-bd24-0628b404b697\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") "
Nov 03 21:16:04.020613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.020491    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\") pod \"f1cc166d-4a1a-4be6-bd24-0628b404b697\" (UID: \"f1cc166d-4a1a-4be6-bd24-0628b404b697\") "
Nov 03 21:16:04.020929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.020899    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume" pod "f1cc166d-4a1a-4be6-bd24-0628b404b697" (UID: "f1cc166d-4a1a-4be6-bd24-0628b404b697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:04.021055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.021037    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1cc166d-4a1a-4be6-bd24-0628b404b697-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:04.022566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.022543    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc" pod "f1cc166d-4a1a-4be6-bd24-0628b404b697" (UID: "f1cc166d-4a1a-4be6-bd24-0628b404b697"). InnerVolumeSpecName "kube-api-access-ps8bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:04.122809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.122781    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8bc\" (UniqueName: \"kubernetes.io/projected/f1cc166d-4a1a-4be6-bd24-0628b404b697-kube-api-access-ps8bc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:04.309053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.309021    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:16:04.313169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.313148    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22600c-75959bb4c-hmdzw"]
Nov 03 21:16:04.494390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:04.494319    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cc166d-4a1a-4be6-bd24-0628b404b697" path="/var/lib/kubelet/pods/f1cc166d-4a1a-4be6-bd24-0628b404b697/volumes"
Nov 03 21:16:19.262455 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:16:19.262417    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:16:19.350490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:19.350465    5533 scope.go:122] "RemoveContainer" containerID="e4a5c757a72b8446cb99949825a82fb618477bef61cac74a9a82c31d6b3529cb"
Nov 03 21:16:19.359575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:19.359545    5533 scope.go:122] "RemoveContainer" containerID="a8dfb154fcf1a52f42edd5206b156c93b840101f502d7a7deabf3fe48874be8c"
Nov 03 21:16:19.368081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:19.368062    5533 scope.go:122] "RemoveContainer" containerID="54da0ba7af28ed68a32a1237e1b77b05cf296325846f0a48ce4ab77a2838a002"
Nov 03 21:16:19.376557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:19.376536    5533 scope.go:122] "RemoveContainer" containerID="934d2a91b7911b4d4eb4d43c143ab7b509ea87273537123cfe59ae0e6200836f"
Nov 03 21:16:26.508140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:26.508107    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:16:35.466788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:35.466750    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:16:35.467372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:35.466818    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:16:37.142107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.142076    5533 generic.go:356] "Generic (PLEG): container finished" podID="e00acc07-571e-43cc-820d-1768a228c626" containerID="ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672" exitCode=0
Nov 03 21:16:37.142565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.142148    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerDied","Data":"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"}
Nov 03 21:16:37.142565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.142504    5533 scope.go:122] "RemoveContainer" containerID="ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"
Nov 03 21:16:37.143883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.143857    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:37.143958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.143921    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:37.143958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:37.143955    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:38.159658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:38.159323    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerStarted","Data":"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"}
Nov 03 21:16:38.913373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:38.913333    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:16:38.913547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:38.913471    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:16:38.913661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:38.913546    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 21:16:39.913882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:39.913848    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:16:39.914363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:39.913919    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 21:16:40.913974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:40.913940    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:16:40.914385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:40.913996    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 21:16:41.913915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:41.913879    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:16:41.914087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:41.913945    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 21:16:42.913285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:42.913249    5533 patch_prober.go:29] interesting pod/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 21:16:42.913459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:42.913305    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 21:16:43.917694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:43.917663    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:16:43.918090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:43.918071    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:16:43.918871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:43.918856    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:16:49.267010 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:16:49.266970    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:16:50.911998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:50.911962    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:16:53.313464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.313429    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:16:53.314159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.314139    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f1cc166d-4a1a-4be6-bd24-0628b404b697" containerName="payload-pre-processing"
Nov 03 21:16:53.314206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.314161    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cc166d-4a1a-4be6-bd24-0628b404b697" containerName="payload-pre-processing"
Nov 03 21:16:53.314300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.314291    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cc166d-4a1a-4be6-bd24-0628b404b697" containerName="payload-pre-processing"
Nov 03 21:16:53.317409 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.317388    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.382683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.368971    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2ad2d-openshift-default-dockercfg-k6ccq\""
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386451    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386512    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386594    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5h4\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386653    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386678    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386724    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386828    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386881    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.387901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.386913    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.405679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.405647    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:16:53.487867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.487823    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.488496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.488473    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.488744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.488726    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5h4\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.488889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.488875    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.488995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.488981    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.489111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.489097    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.489236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.489221    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.489348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.489336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.489453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.489441    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.490118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.490096    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.490243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.488413    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.491615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.491594    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.491983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.491831    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.492330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.492310    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.493091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.493070    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.494713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.494691    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.569371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.569299    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5h4\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.603370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.603303    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\") pod \"e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") " pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:53.630485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:53.630437    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:54.099656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.099313    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:16:54.120973 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:16:54.116835    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a72afed_8dab_4432_8ab3_f1fde1647085.slice/crio-605e135464d3a48de21791fa221887d0a920e99dfb0e814e7de133ff13caf2e8 WatchSource:0}: Error finding container 605e135464d3a48de21791fa221887d0a920e99dfb0e814e7de133ff13caf2e8: Status 404 returned error can't find the container with id 605e135464d3a48de21791fa221887d0a920e99dfb0e814e7de133ff13caf2e8
Nov 03 21:16:54.120973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.119942    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:54.120973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.120027    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:54.120973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.120073    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:16:54.316796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.316755    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerStarted","Data":"db6b62367bf7c46e591b73bfb20ea61e4181ac814aa5274acbcc810967498649"}
Nov 03 21:16:54.317199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.316854    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerStarted","Data":"605e135464d3a48de21791fa221887d0a920e99dfb0e814e7de133ff13caf2e8"}
Nov 03 21:16:54.432178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.432130    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podStartSLOduration=1.432117281 podStartE2EDuration="1.432117281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:54.43038247 +0000 UTC m=+1716.061695940" watchObservedRunningTime="2026-11-03 21:16:54.432117281 +0000 UTC m=+1716.063430757"
Nov 03 21:16:54.631644 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:54.631575    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:55.636573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:55.636540    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:55.912708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:55.912585    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" podUID="8e3f7bb6-a586-480c-9fba-456531e18224" containerName="payload-pre-processing" containerID="cri-o://9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652" gracePeriod=25
Nov 03 21:16:56.183575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.183548    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:16:56.332929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.332891    5533 generic.go:356] "Generic (PLEG): container finished" podID="8e3f7bb6-a586-480c-9fba-456531e18224" containerID="9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652" exitCode=0
Nov 03 21:16:56.333110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.332942    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"
Nov 03 21:16:56.333110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.332985    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" event={"ID":"8e3f7bb6-a586-480c-9fba-456531e18224","Type":"ContainerDied","Data":"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"}
Nov 03 21:16:56.333110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.333054    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p" event={"ID":"8e3f7bb6-a586-480c-9fba-456531e18224","Type":"ContainerDied","Data":"775bbba79fefc1569ded9d86cdd0aa7b211fcaeade71785e5d76a84d08ade5cb"}
Nov 03 21:16:56.333110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.333089    5533 scope.go:122] "RemoveContainer" containerID="9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"
Nov 03 21:16:56.333555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.333532    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:56.334625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.334606    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:16:56.338395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.338370    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\" (UniqueName: \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\") pod \"8e3f7bb6-a586-480c-9fba-456531e18224\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") "
Nov 03 21:16:56.338549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.338530    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\") pod \"8e3f7bb6-a586-480c-9fba-456531e18224\" (UID: \"8e3f7bb6-a586-480c-9fba-456531e18224\") "
Nov 03 21:16:56.339164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.339109    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume" pod "8e3f7bb6-a586-480c-9fba-456531e18224" (UID: "8e3f7bb6-a586-480c-9fba-456531e18224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:56.340678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.340657    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26" pod "8e3f7bb6-a586-480c-9fba-456531e18224" (UID: "8e3f7bb6-a586-480c-9fba-456531e18224"). InnerVolumeSpecName "kube-api-access-lwk26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:56.346355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.346333    5533 scope.go:122] "RemoveContainer" containerID="9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"
Nov 03 21:16:56.346682 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:16:56.346654    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652\": container with ID starting with 9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652 not found: ID does not exist" containerID="9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"
Nov 03 21:16:56.346781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.346692    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652"} err="failed to get container status \"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652\": rpc error: code = NotFound desc = could not find container \"9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652\": container with ID starting with 9f5eda22970328fdc9440765d0dda01ddae04b075bed17aafa1eecab86d48652 not found: ID does not exist"
Nov 03 21:16:56.440228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.440165    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwk26\" (UniqueName: \"kubernetes.io/projected/8e3f7bb6-a586-480c-9fba-456531e18224-kube-api-access-lwk26\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:56.440228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.440194    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3f7bb6-a586-480c-9fba-456531e18224-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:56.743320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.743123    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:16:56.747180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:56.747155    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812243-bcb58d59-zmf2p"]
Nov 03 21:16:58.504163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:58.504063    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3f7bb6-a586-480c-9fba-456531e18224" path="/var/lib/kubelet/pods/8e3f7bb6-a586-480c-9fba-456531e18224/volumes"
Nov 03 21:16:58.720669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:16:58.720606    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:17:05.896421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:05.896380    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:17:06.261376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:06.261337    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:17:06.261615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:06.261585    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" containerID="cri-o://f6f1b78828babc6d91c62457fd78f766affe2da74bc4681c7b9c793ec41daf88" gracePeriod=30
Nov 03 21:17:07.892401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:07.892356    5533 patch_prober.go:29] interesting pod/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:17:07.893005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:07.892958    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:17:11.492150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.492110    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" podUID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" containerName="payload-processing" containerID="cri-o://09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730" gracePeriod=25
Nov 03 21:17:11.527773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.527718    5533 generic.go:356] "Generic (PLEG): container finished" podID="f4f17222-e41f-4f14-9917-57c949467fe6" containerID="f6f1b78828babc6d91c62457fd78f766affe2da74bc4681c7b9c793ec41daf88" exitCode=0
Nov 03 21:17:11.527899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.527790    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerDied","Data":"f6f1b78828babc6d91c62457fd78f766affe2da74bc4681c7b9c793ec41daf88"}
Nov 03 21:17:11.527899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.527854    5533 scope.go:122] "RemoveContainer" containerID="146a9d322d3a2bcf0932555b3a54189c8c179209ae32ed28fa008bcf6003c9c1"
Nov 03 21:17:11.626904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.626879    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:17:11.671468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671438    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.671468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671478    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671550    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671604    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671749    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671806    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671817    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671914    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.671945    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.672000    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\") pod \"f4f17222-e41f-4f14-9917-57c949467fe6\" (UID: \"f4f17222-e41f-4f14-9917-57c949467fe6\") "
Nov 03 21:17:11.672689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.672624    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.673051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.673013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:11.673308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.673285    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:11.673451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.673432    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:11.673670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.673629    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:11.675103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.675059    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "kube-api-access-7whnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:11.676752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.676724    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:11.688562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.688540    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:11.689732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.689707    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token" pod "f4f17222-e41f-4f14-9917-57c949467fe6" (UID: "f4f17222-e41f-4f14-9917-57c949467fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:11.774332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774300    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774350    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7whnl\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-kube-api-access-7whnl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774363    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774372    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774381    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f17222-e41f-4f14-9917-57c949467fe6-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774390    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f17222-e41f-4f14-9917-57c949467fe6-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774398    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f17222-e41f-4f14-9917-57c949467fe6-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.774465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.774406    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f17222-e41f-4f14-9917-57c949467fe6-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.782322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.782269    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:17:11.875360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.875334    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\" (UniqueName: \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\") pod \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") "
Nov 03 21:17:11.875496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.875379    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\") pod \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\" (UID: \"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8\") "
Nov 03 21:17:11.875776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.875751    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume" pod "73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" (UID: "73e0fb5a-8902-4418-ad6f-198c0e3ed2c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:11.877400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.877382    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl" pod "73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" (UID: "73e0fb5a-8902-4418-ad6f-198c0e3ed2c8"). InnerVolumeSpecName "kube-api-access-9ljbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:11.977497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.977457    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ljbl\" (UniqueName: \"kubernetes.io/projected/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-kube-api-access-9ljbl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:11.977497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:11.977498    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:12.536545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.536483    5533 generic.go:356] "Generic (PLEG): container finished" podID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" containerID="09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730" exitCode=0
Nov 03 21:17:12.536545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.536528    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"
Nov 03 21:17:12.537032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.536558    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" event={"ID":"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8","Type":"ContainerDied","Data":"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"}
Nov 03 21:17:12.537032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.536608    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn" event={"ID":"73e0fb5a-8902-4418-ad6f-198c0e3ed2c8","Type":"ContainerDied","Data":"8577d1b8e313bd9b5491205fdaf8e483d30ca9d31c08001a91f8d5837ced1507"}
Nov 03 21:17:12.537032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.536630    5533 scope.go:122] "RemoveContainer" containerID="09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"
Nov 03 21:17:12.538364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.538335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr" event={"ID":"f4f17222-e41f-4f14-9917-57c949467fe6","Type":"ContainerDied","Data":"101b78ff48fad05727700c349833c54fbe69f9a33f70749d81f3b686a8b54c98"}
Nov 03 21:17:12.538480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.538341    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"
Nov 03 21:17:12.547260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.547243    5533 scope.go:122] "RemoveContainer" containerID="09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"
Nov 03 21:17:12.547558 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:17:12.547541    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730\": container with ID starting with 09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730 not found: ID does not exist" containerID="09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"
Nov 03 21:17:12.547630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.547569    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730"} err="failed to get container status \"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730\": rpc error: code = NotFound desc = could not find container \"09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730\": container with ID starting with 09f13db9e885d803425ee86b40b55e200f845a566789370cd4967abe63c32730 not found: ID does not exist"
Nov 03 21:17:12.547630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.547591    5533 scope.go:122] "RemoveContainer" containerID="f6f1b78828babc6d91c62457fd78f766affe2da74bc4681c7b9c793ec41daf88"
Nov 03 21:17:12.768468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.768437    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:17:12.783419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.783390    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-22600c-openshift-default-7f95fc48d9-wf8pr"]
Nov 03 21:17:12.890643 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.890562    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:17:12.893885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:12.893863    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-32f6e43c-fc4b7f84d-fxfvn"]
Nov 03 21:17:14.104440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:14.104409    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:17:14.495803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:14.495769    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" path="/var/lib/kubelet/pods/73e0fb5a-8902-4418-ad6f-198c0e3ed2c8/volumes"
Nov 03 21:17:14.496273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:14.496256    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" path="/var/lib/kubelet/pods/f4f17222-e41f-4f14-9917-57c949467fe6/volumes"
Nov 03 21:17:18.856393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.856358    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:17:18.857134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857110    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857146    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857185    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857193    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857212    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8e3f7bb6-a586-480c-9fba-456531e18224" containerName="payload-pre-processing"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857220    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3f7bb6-a586-480c-9fba-456531e18224" containerName="payload-pre-processing"
Nov 03 21:17:18.857246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857242    5533 cpu_manager.go:414] "containerMap: removing container" podUID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" containerName="payload-processing"
Nov 03 21:17:18.857561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857250    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" containerName="payload-processing"
Nov 03 21:17:18.857561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857374    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:18.857561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857394    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3f7bb6-a586-480c-9fba-456531e18224" containerName="payload-pre-processing"
Nov 03 21:17:18.857561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.857406    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e0fb5a-8902-4418-ad6f-198c0e3ed2c8" containerName="payload-processing"
Nov 03 21:17:18.862488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.862463    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:18.911792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.911755    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eae138\""
Nov 03 21:17:18.912330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.912306    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eae138-dockercfg-r8zbh\""
Nov 03 21:17:18.917102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.917073    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:17:18.962441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.962411    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:18.962613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:18.962469    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2xl\" (UniqueName: \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.032803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.032771    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:17:19.033402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.033380    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f17222-e41f-4f14-9917-57c949467fe6" containerName="istio-proxy"
Nov 03 21:17:19.036691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.036668    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.063381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.063355    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.063465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.063407    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2xl\" (UniqueName: \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.064001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.063984    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.105057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.104971    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" podUID="9b9235ac-b0e7-4701-98a7-121e1beb1310" containerName="payload-processing" containerID="cri-o://adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13" gracePeriod=25
Nov 03 21:17:19.111100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.111038    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:17:19.122585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.122561    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2xl\" (UniqueName: \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\") pod \"payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.164537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.164511    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.164653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.164619    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8ws\" (UniqueName: \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.193266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.193240    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:19.269427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.267701    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.269427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.267842    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8ws\" (UniqueName: \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.269427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.269106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.279673 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:17:19.277262    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:17:19.328940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.328892    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8ws\" (UniqueName: \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\") pod \"payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.350829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.350801    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:19.412714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.412678    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:17:19.567327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.567294    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:17:19.567967 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:17:19.567938    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa9b36b_9b0c_4562_942f_a425c4665edd.slice/crio-c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd WatchSource:0}: Error finding container c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd: Status 404 returned error can't find the container with id c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd
Nov 03 21:17:19.573554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.573525    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\") pod \"9b9235ac-b0e7-4701-98a7-121e1beb1310\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") "
Nov 03 21:17:19.573733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.573714    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\") pod \"9b9235ac-b0e7-4701-98a7-121e1beb1310\" (UID: \"9b9235ac-b0e7-4701-98a7-121e1beb1310\") "
Nov 03 21:17:19.574129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.574099    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume" pod "9b9235ac-b0e7-4701-98a7-121e1beb1310" (UID: "9b9235ac-b0e7-4701-98a7-121e1beb1310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:19.574312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.574289    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9235ac-b0e7-4701-98a7-121e1beb1310-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:19.576404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.576373    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs" pod "9b9235ac-b0e7-4701-98a7-121e1beb1310" (UID: "9b9235ac-b0e7-4701-98a7-121e1beb1310"). InnerVolumeSpecName "kube-api-access-fbdxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:19.589203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.589179    5533 generic.go:356] "Generic (PLEG): container finished" podID="9b9235ac-b0e7-4701-98a7-121e1beb1310" containerID="adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13" exitCode=0
Nov 03 21:17:19.589319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.589223    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"
Nov 03 21:17:19.589319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.589259    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" event={"ID":"9b9235ac-b0e7-4701-98a7-121e1beb1310","Type":"ContainerDied","Data":"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"}
Nov 03 21:17:19.589319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.589306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6" event={"ID":"9b9235ac-b0e7-4701-98a7-121e1beb1310","Type":"ContainerDied","Data":"24b304007bcaafcc4ee5ef65d667de2207df03ef1084c85a5150dca6df51f835"}
Nov 03 21:17:19.589483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.589329    5533 scope.go:122] "RemoveContainer" containerID="adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"
Nov 03 21:17:19.591030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.591001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" event={"ID":"8aa9b36b-9b0c-4562-942f-a425c4665edd","Type":"ContainerStarted","Data":"c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd"}
Nov 03 21:17:19.600858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.600837    5533 scope.go:122] "RemoveContainer" containerID="adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"
Nov 03 21:17:19.601192 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:17:19.601165    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13\": container with ID starting with adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13 not found: ID does not exist" containerID="adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"
Nov 03 21:17:19.601249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.601203    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13"} err="failed to get container status \"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13\": rpc error: code = NotFound desc = could not find container \"adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13\": container with ID starting with adb093be90d2ee0e055c86f3dc52809394b4a526586c19e0f263f30f52e68b13 not found: ID does not exist"
Nov 03 21:17:19.675451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.675422    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdxs\" (UniqueName: \"kubernetes.io/projected/9b9235ac-b0e7-4701-98a7-121e1beb1310-kube-api-access-fbdxs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:19.774248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.774220    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:17:19.774807 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:17:19.774781    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aabb67_b543_4907_892f_c5196eade0c4.slice/crio-7e03f64392b292e39e2e8b7940336387ae9034e5f3964397dd0e49a88ca8981a WatchSource:0}: Error finding container 7e03f64392b292e39e2e8b7940336387ae9034e5f3964397dd0e49a88ca8981a: Status 404 returned error can't find the container with id 7e03f64392b292e39e2e8b7940336387ae9034e5f3964397dd0e49a88ca8981a
Nov 03 21:17:19.778204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.778180    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:17:19.785777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:19.785755    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-66513c-5b67dbd968-sxnc6"]
Nov 03 21:17:20.496143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.496111    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9235ac-b0e7-4701-98a7-121e1beb1310" path="/var/lib/kubelet/pods/9b9235ac-b0e7-4701-98a7-121e1beb1310/volumes"
Nov 03 21:17:20.600255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.600223    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" event={"ID":"86aabb67-b543-4907-892f-c5196eade0c4","Type":"ContainerStarted","Data":"bbc1f2ff7f4959139d144498a02a9d2c88273290394a324cadcfecd2fc099ec5"}
Nov 03 21:17:20.600426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.600281    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" event={"ID":"86aabb67-b543-4907-892f-c5196eade0c4","Type":"ContainerStarted","Data":"7e03f64392b292e39e2e8b7940336387ae9034e5f3964397dd0e49a88ca8981a"}
Nov 03 21:17:20.600483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.600431    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:20.602007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.601983    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" event={"ID":"8aa9b36b-9b0c-4562-942f-a425c4665edd","Type":"ContainerStarted","Data":"c79da4ba0eb710d4c9e6ab495b9754423a53fafbb9577f252874efca5b360462"}
Nov 03 21:17:20.602128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.602096    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:20.713225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.713184    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" podStartSLOduration=2.713173004 podStartE2EDuration="2.713173004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:20.709784469 +0000 UTC m=+1742.341097945" watchObservedRunningTime="2026-11-03 21:17:20.713173004 +0000 UTC m=+1742.344486495"
Nov 03 21:17:20.820769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:20.820692    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" podStartSLOduration=2.820680494 podStartE2EDuration="2.820680494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:20.819256361 +0000 UTC m=+1742.450569837" watchObservedRunningTime="2026-11-03 21:17:20.820680494 +0000 UTC m=+1742.451993970"
Nov 03 21:17:25.722621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:25.722579    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:17:31.612277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:31.612246    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:17:31.612676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:31.612321    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:17:41.046982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:41.046948    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"]
Nov 03 21:17:41.093591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:41.093562    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e17c2-fb5hb"]
Nov 03 21:17:42.494710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:42.494677    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611414be-1555-491d-b640-392e6c6c0f48" path="/var/lib/kubelet/pods/611414be-1555-491d-b640-392e6c6c0f48/volumes"
Nov 03 21:17:46.893171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:46.893130    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:17:49.261230 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:17:49.261193    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:17:51.150897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.150859    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"]
Nov 03 21:17:51.151508 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.151483    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9b9235ac-b0e7-4701-98a7-121e1beb1310" containerName="payload-processing"
Nov 03 21:17:51.151508 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.151506    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9235ac-b0e7-4701-98a7-121e1beb1310" containerName="payload-processing"
Nov 03 21:17:51.151787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.151654    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9235ac-b0e7-4701-98a7-121e1beb1310" containerName="payload-processing"
Nov 03 21:17:51.155680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.155660    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.206546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.206520    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:17:51.253007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.252978    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.253157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.253036    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tt76\" (UniqueName: \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.253240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.253166    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.253293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.253238    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:17:51.256923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.256901    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"]
Nov 03 21:17:51.354292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.354267    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.354422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.354307    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tt76\" (UniqueName: \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.354492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.354426    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.354980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.354959    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.356977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.356943    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.411912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.411851    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tt76\" (UniqueName: \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\") pod \"maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.470967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.470946    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:51.842584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.842553    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"]
Nov 03 21:17:51.844268 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:17:51.844229    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d7444a_0adc_45eb_b392_906070183a9c.slice/crio-3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e WatchSource:0}: Error finding container 3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e: Status 404 returned error can't find the container with id 3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e
Nov 03 21:17:51.899672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:51.898698    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" podUID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" containerName="payload-pre-processing" containerID="cri-o://9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b" gracePeriod=25
Nov 03 21:17:52.173833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.173808    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:17:52.267815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.267791    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\" (UniqueName: \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\") pod \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") "
Nov 03 21:17:52.267967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.267874    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\") pod \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\" (UID: \"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec\") "
Nov 03 21:17:52.268246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.268225    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume" pod "c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" (UID: "c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:52.268465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.268447    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:52.270009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.269987    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6" pod "c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" (UID: "c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec"). InnerVolumeSpecName "kube-api-access-9ctb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:52.370284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.370252    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctb6\" (UniqueName: \"kubernetes.io/projected/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec-kube-api-access-9ctb6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:52.841530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.841504    5533 generic.go:356] "Generic (PLEG): container finished" podID="25d7444a-0adc-45eb-b392-906070183a9c" containerID="4ff4d261e0c7fd2ccbb8c041a816fa46d9964a911451ed1575953a4ea83c5710" exitCode=0
Nov 03 21:17:52.841675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.841578    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9" event={"ID":"25d7444a-0adc-45eb-b392-906070183a9c","Type":"ContainerDied","Data":"4ff4d261e0c7fd2ccbb8c041a816fa46d9964a911451ed1575953a4ea83c5710"}
Nov 03 21:17:52.841675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.841631    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9" event={"ID":"25d7444a-0adc-45eb-b392-906070183a9c","Type":"ContainerStarted","Data":"3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e"}
Nov 03 21:17:52.842910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.842889    5533 generic.go:356] "Generic (PLEG): container finished" podID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" containerID="9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b" exitCode=0
Nov 03 21:17:52.843022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.842928    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"
Nov 03 21:17:52.843022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.842971    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" event={"ID":"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec","Type":"ContainerDied","Data":"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"}
Nov 03 21:17:52.843022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.843021    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x" event={"ID":"c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec","Type":"ContainerDied","Data":"96826dbbcb19077dc3b6743c6f77ea0421f955546736a2ca960eabccbf92eed4"}
Nov 03 21:17:52.843182 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.843043    5533 scope.go:122] "RemoveContainer" containerID="9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"
Nov 03 21:17:52.884609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.884590    5533 scope.go:122] "RemoveContainer" containerID="9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"
Nov 03 21:17:52.884956 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:17:52.884931    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b\": container with ID starting with 9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b not found: ID does not exist" containerID="9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"
Nov 03 21:17:52.885057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:52.884966    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b"} err="failed to get container status \"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b\": rpc error: code = NotFound desc = could not find container \"9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b\": container with ID starting with 9fc4a84fa23d9f395f1c09ffcfaca1ef413244989e0d8a1ba07f716d7e5eab5b not found: ID does not exist"
Nov 03 21:17:53.057840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:53.057804    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:17:53.063397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:53.063371    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9232e5-7df4f7b54c-ntf8x"]
Nov 03 21:17:53.998956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:53.998922    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:54.188224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.188197    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\" (UniqueName: \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\") pod \"25d7444a-0adc-45eb-b392-906070183a9c\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") "
Nov 03 21:17:54.188378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.188305    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\") pod \"25d7444a-0adc-45eb-b392-906070183a9c\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") "
Nov 03 21:17:54.188378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.188372    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\") pod \"25d7444a-0adc-45eb-b392-906070183a9c\" (UID: \"25d7444a-0adc-45eb-b392-906070183a9c\") "
Nov 03 21:17:54.188716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.188671    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca" pod "25d7444a-0adc-45eb-b392-906070183a9c" (UID: "25d7444a-0adc-45eb-b392-906070183a9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:54.188845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.188830    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.190451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.190430    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca" pod "25d7444a-0adc-45eb-b392-906070183a9c" (UID: "25d7444a-0adc-45eb-b392-906070183a9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:54.190664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.190624    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76" pod "25d7444a-0adc-45eb-b392-906070183a9c" (UID: "25d7444a-0adc-45eb-b392-906070183a9c"). InnerVolumeSpecName "kube-api-access-5tt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:54.289556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.289525    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25d7444a-0adc-45eb-b392-906070183a9c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.289556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.289553    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tt76\" (UniqueName: \"kubernetes.io/projected/25d7444a-0adc-45eb-b392-906070183a9c-kube-api-access-5tt76\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.493709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.493656    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" path="/var/lib/kubelet/pods/c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec/volumes"
Nov 03 21:17:54.857666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.857578    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"
Nov 03 21:17:54.857666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.857612    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9" event={"ID":"25d7444a-0adc-45eb-b392-906070183a9c","Type":"ContainerDied","Data":"3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e"}
Nov 03 21:17:54.857850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:17:54.857689    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3505d95e68b7b4cdff10a1a89e2b66dfd98ced3a3d568f52772ea2e902cc985e"
Nov 03 21:18:05.946490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.946454    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:18:05.947024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947009    5533 cpu_manager.go:414] "containerMap: removing container" podUID="25d7444a-0adc-45eb-b392-906070183a9c" containerName="revoke-keys"
Nov 03 21:18:05.947082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947026    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d7444a-0adc-45eb-b392-906070183a9c" containerName="revoke-keys"
Nov 03 21:18:05.947082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947057    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" containerName="payload-pre-processing"
Nov 03 21:18:05.947082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947065    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" containerName="payload-pre-processing"
Nov 03 21:18:05.947211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947197    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d7444a-0adc-45eb-b392-906070183a9c" containerName="revoke-keys"
Nov 03 21:18:05.947246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.947220    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9495d0d-e3cb-4dd0-bce7-1e70ce1bd8ec" containerName="payload-pre-processing"
Nov 03 21:18:05.950972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:05.950950    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.029699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.026609    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.029699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.026854    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mcr\" (UniqueName: \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.048379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.048333    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-66513c\""
Nov 03 21:18:06.048557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.048492    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-66513c-dockercfg-wd8bs\""
Nov 03 21:18:06.052673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.052649    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:18:06.127811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.127780    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.127932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.127902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mcr\" (UniqueName: \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.128496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.128478    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.185604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.185579    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mcr\" (UniqueName: \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\") pod \"payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.265442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.265372    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:06.639542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.639515    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:18:06.641112 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:18:06.641066    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea0d2ed_44ad_4b68_af03_77e134f670ec.slice/crio-4bfab5b7f6a70206e4576e50801c03f01702e6e07de0bde5222c012736f8f453 WatchSource:0}: Error finding container 4bfab5b7f6a70206e4576e50801c03f01702e6e07de0bde5222c012736f8f453: Status 404 returned error can't find the container with id 4bfab5b7f6a70206e4576e50801c03f01702e6e07de0bde5222c012736f8f453
Nov 03 21:18:06.643024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.643005    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:18:06.937143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:06.937107    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" event={"ID":"aea0d2ed-44ad-4b68-af03-77e134f670ec","Type":"ContainerStarted","Data":"4bfab5b7f6a70206e4576e50801c03f01702e6e07de0bde5222c012736f8f453"}
Nov 03 21:18:09.964220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:09.964182    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" event={"ID":"aea0d2ed-44ad-4b68-af03-77e134f670ec","Type":"ContainerStarted","Data":"c4b1d143843f1dc3702ca78d10d916c231237fdb0bba1ac1e849a5a0f7461730"}
Nov 03 21:18:09.964623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:09.964275    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:10.073753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:10.073696    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" podStartSLOduration=2.623168119 podStartE2EDuration="5.073673716s" totalImagesPullingTime="2.450505597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:10.073355346 +0000 UTC m=+1791.704668826" watchObservedRunningTime="2026-11-03 21:18:10.073673716 +0000 UTC m=+1791.704987194"
Nov 03 21:18:11.279315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:11.279282    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:18:18.766458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:18.766422    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:18:18.776432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:18.776400    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:18:18.784139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:18.784114    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:18:18.789068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:18.789046    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:18:19.265835 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:18:19.265795    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:18:19.553433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:19.553360    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:18:19.553922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:19.553624    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy" containerID="cri-o://2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55" gracePeriod=30
Nov 03 21:18:20.979349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:20.979321    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:18:22.046421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.046380    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"]
Nov 03 21:18:22.051886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.051860    5533 generic.go:356] "Generic (PLEG): container finished" podID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerID="db6b62367bf7c46e591b73bfb20ea61e4181ac814aa5274acbcc810967498649" exitCode=0
Nov 03 21:18:22.052040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.051934    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerDied","Data":"db6b62367bf7c46e591b73bfb20ea61e4181ac814aa5274acbcc810967498649"}
Nov 03 21:18:22.052320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.052303    5533 scope.go:122] "RemoveContainer" containerID="db6b62367bf7c46e591b73bfb20ea61e4181ac814aa5274acbcc810967498649"
Nov 03 21:18:22.053702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.053675    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:22.053795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.053741    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:22.053795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.053770    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:22.092989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.092962    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c80f6b5-wcp9n"]
Nov 03 21:18:22.495163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:22.495128    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0218c44-799a-4d03-8bbb-e36eaf5ecd35" path="/var/lib/kubelet/pods/e0218c44-799a-4d03-8bbb-e36eaf5ecd35/volumes"
Nov 03 21:18:23.110318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.110279    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerStarted","Data":"261f0ba8a8ee549ec3ae5e9e0c7d4958d1f426803e1d21dc36783d5f7bb65d77"}
Nov 03 21:18:23.631061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.631030    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:18:23.632067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.632041    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 21:18:23.632171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.632091    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 21:18:23.781334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.781290    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:18:23.788812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.788787    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.835252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.835211    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:18:23.838268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.838243    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-812ef9-openshift-default-dockercfg-lv9nv\""
Nov 03 21:18:23.908617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908543    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.908788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908617    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.908788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908661    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.908788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908775    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.908901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908806    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.908901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.908888    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.909041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.909025    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.909078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.909049    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:23.909110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:23.909086    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhdr\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.010765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.010733    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.010977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.010867    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.010977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.010893    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.010977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.010934    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhdr\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.010990    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011023    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011057    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011145    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011181    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011210    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011350    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011522    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011549    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.011885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.011867    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.013224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.013200    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.013430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.013411    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.076999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.076971    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.077197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.077171    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhdr\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\") pod \"e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") " pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.104738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.104715    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:24.577311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.570271    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:18:24.578342 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:18:24.578288    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44557d39_db06_4b1c_b42a_8168134023b5.slice/crio-c1e0f349dbf318ca878f563aaaea6df746ee518d79ce4ae6328e35bdd2d2026c WatchSource:0}: Error finding container c1e0f349dbf318ca878f563aaaea6df746ee518d79ce4ae6328e35bdd2d2026c: Status 404 returned error can't find the container with id c1e0f349dbf318ca878f563aaaea6df746ee518d79ce4ae6328e35bdd2d2026c
Nov 03 21:18:24.580932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.580900    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:24.581014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.580965    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:24.581014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.580996    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:24.630978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.630956    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 21:18:24.631073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.630998    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 21:18:24.842574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.842547    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:18:24.922529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922492    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.922722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922559    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.922722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922596    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.922722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922618    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.922722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922657    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.922722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922710    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.923041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922746    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.923041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922764    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.923041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922781    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\") pod \"bf8161d5-9550-40b0-acc2-96b7040adb07\" (UID: \"bf8161d5-9550-40b0-acc2-96b7040adb07\") "
Nov 03 21:18:24.923041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.922939    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:18:24.923249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.923146    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8161d5-9550-40b0-acc2-96b7040adb07-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:24.923533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.923510    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:24.923890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.923490    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:24.923890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.923538    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:24.923890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.923734    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:24.925470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.925444    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:24.925700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.925510    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:24.925791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.925721    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:18:24.925948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:24.925930    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq" pod "bf8161d5-9550-40b0-acc2-96b7040adb07" (UID: "bf8161d5-9550-40b0-acc2-96b7040adb07"). InnerVolumeSpecName "kube-api-access-t7dgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:25.024592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024563    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024602    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024618    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024645    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024661    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8161d5-9550-40b0-acc2-96b7040adb07-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024675    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024689    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.024846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.024703    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7dgq\" (UniqueName: \"kubernetes.io/projected/bf8161d5-9550-40b0-acc2-96b7040adb07-kube-api-access-t7dgq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:18:25.134050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.133960    5533 generic.go:356] "Generic (PLEG): container finished" podID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerID="2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55" exitCode=0
Nov 03 21:18:25.134050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.133998    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerDied","Data":"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"}
Nov 03 21:18:25.134050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.134022    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"
Nov 03 21:18:25.134318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.134067    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h" event={"ID":"bf8161d5-9550-40b0-acc2-96b7040adb07","Type":"ContainerDied","Data":"30f37dd9abc5f79cc8cfccd92618dd57492fe6f50edc62cd23a64f1be4c0bead"}
Nov 03 21:18:25.134318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.134097    5533 scope.go:122] "RemoveContainer" containerID="2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"
Nov 03 21:18:25.138677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.138631    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerStarted","Data":"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"}
Nov 03 21:18:25.138762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.138687    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerStarted","Data":"c1e0f349dbf318ca878f563aaaea6df746ee518d79ce4ae6328e35bdd2d2026c"}
Nov 03 21:18:25.148688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.148669    5533 scope.go:122] "RemoveContainer" containerID="f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.185210    5533 scope.go:122] "RemoveContainer" containerID="2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:18:25.185667    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55\": container with ID starting with 2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55 not found: ID does not exist" containerID="2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.185702    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55"} err="failed to get container status \"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55\": rpc error: code = NotFound desc = could not find container \"2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55\": container with ID starting with 2f57261c208013676d7aa456f4c1507542d7be907181eea59906898b35764d55 not found: ID does not exist"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.185738    5533 scope.go:122] "RemoveContainer" containerID="f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:18:25.186078    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89\": container with ID starting with f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89 not found: ID does not exist" containerID="f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"
Nov 03 21:18:25.187837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.186106    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89"} err="failed to get container status \"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89\": rpc error: code = NotFound desc = could not find container \"f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89\": container with ID starting with f5a0b8dc2cddedd8d95ad9a0d8bf068b855a17112cc8c23e1bc7af2160bafb89 not found: ID does not exist"
Nov 03 21:18:25.295763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.295488    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:18:25.300400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.300369    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0f8726-openshift-default-fd579cb7c-ps52h"]
Nov 03 21:18:25.406945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.406829    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podStartSLOduration=2.406815912 podStartE2EDuration="2.406815912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:25.405837296 +0000 UTC m=+1807.037150773" watchObservedRunningTime="2026-11-03 21:18:25.406815912 +0000 UTC m=+1807.038129452"
Nov 03 21:18:25.630876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.630845    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 21:18:25.631233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:25.630904    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 21:18:26.105383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.105349    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:26.111187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.111163    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:26.182659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.179307    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:26.183877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.183852    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:18:26.334366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.334335    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:18:26.496820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.496785    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" path="/var/lib/kubelet/pods/bf8161d5-9550-40b0-acc2-96b7040adb07/volumes"
Nov 03 21:18:26.631246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.631218    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 21:18:26.631682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:26.631268    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 21:18:27.631183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:27.631152    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 21:18:27.631423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:27.631394    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 21:18:28.636063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:28.636034    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:18:28.637138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:28.637118    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:18:31.888457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.888418    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:18:31.889270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.889250    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:31.889326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.889274    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:31.889326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.889297    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:31.889326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.889305    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:31.889442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.889430    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:31.900824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.900801    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:31.947282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.947256    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:18:31.989612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:31.989575    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-812ef9\""
Nov 03 21:18:32.014091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.014061    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-812ef9-dockercfg-jq456\""
Nov 03 21:18:32.022972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.022937    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnmw\" (UniqueName: \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.023095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.023062    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.037463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.037435    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:18:32.039702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.039679    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8161d5-9550-40b0-acc2-96b7040adb07" containerName="istio-proxy"
Nov 03 21:18:32.060829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.060794    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.128358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.126838    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.128358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.126976    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff56s\" (UniqueName: \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.128358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.127097    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnmw\" (UniqueName: \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.128358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.127142    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.128358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.128055    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.179957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.179912    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:18:32.230279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.230240    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnmw\" (UniqueName: \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\") pod \"payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.232215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.232183    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.232369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.232354    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff56s\" (UniqueName: \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.233517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.233473    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.295682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.295628    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff56s\" (UniqueName: \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\") pod \"payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.453350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.453264    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:32.514184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.514160    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:32.831928 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:18:32.831891    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79ce08f_f101_44c9_ab8e_597c8b5e4adf.slice/crio-9f241907078456bf027cf76cf3f4cffc79bd12adad9ebb5411410bbd0d4e4794 WatchSource:0}: Error finding container 9f241907078456bf027cf76cf3f4cffc79bd12adad9ebb5411410bbd0d4e4794: Status 404 returned error can't find the container with id 9f241907078456bf027cf76cf3f4cffc79bd12adad9ebb5411410bbd0d4e4794
Nov 03 21:18:32.834727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.832821    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:18:32.939835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:32.939807    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:18:32.941809 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:18:32.941773    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e72669b_fc4e_412a_aa3c_eccc975ec9c4.slice/crio-a96f11d5bfa91aeb254daa164ed4d13311caa9b71feea6194b03f9e5af313239 WatchSource:0}: Error finding container a96f11d5bfa91aeb254daa164ed4d13311caa9b71feea6194b03f9e5af313239: Status 404 returned error can't find the container with id a96f11d5bfa91aeb254daa164ed4d13311caa9b71feea6194b03f9e5af313239
Nov 03 21:18:33.247666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.247612    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" event={"ID":"0e72669b-fc4e-412a-aa3c-eccc975ec9c4","Type":"ContainerStarted","Data":"e597fca9b34fb6287b7089211c0278d37398704b80ae279054cd127cbe7132db"}
Nov 03 21:18:33.247844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.247693    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" event={"ID":"0e72669b-fc4e-412a-aa3c-eccc975ec9c4","Type":"ContainerStarted","Data":"a96f11d5bfa91aeb254daa164ed4d13311caa9b71feea6194b03f9e5af313239"}
Nov 03 21:18:33.247844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.247716    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:33.249308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.249285    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" event={"ID":"a79ce08f-f101-44c9-ab8e-597c8b5e4adf","Type":"ContainerStarted","Data":"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"}
Nov 03 21:18:33.249427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.249332    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" event={"ID":"a79ce08f-f101-44c9-ab8e-597c8b5e4adf","Type":"ContainerStarted","Data":"9f241907078456bf027cf76cf3f4cffc79bd12adad9ebb5411410bbd0d4e4794"}
Nov 03 21:18:33.249427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.249363    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:33.364490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.364441    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" podStartSLOduration=2.364428313 podStartE2EDuration="2.364428313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:33.361771099 +0000 UTC m=+1814.993084579" watchObservedRunningTime="2026-11-03 21:18:33.364428313 +0000 UTC m=+1814.995741803"
Nov 03 21:18:33.471942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:33.471902    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" podStartSLOduration=2.47188922 podStartE2EDuration="2.47188922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:33.470512204 +0000 UTC m=+1815.101825681" watchObservedRunningTime="2026-11-03 21:18:33.47188922 +0000 UTC m=+1815.103202697"
Nov 03 21:18:41.372240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.372199    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:18:41.377748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.377716    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.431659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.431155    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3e1b49-openshift-default-dockercfg-cm69g\""
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.467967    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468060    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468117    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468200    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468254    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468343    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468368    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468422    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbw7\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.475694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.468464    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.512288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.512253    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:18:41.573976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.573937    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.573976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.573987    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574039    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbw7\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574080    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574140    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574215    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574270    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574363    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574425    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574825    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574854    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.574980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.574928    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.575223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.575094    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.575591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.575561    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.578403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.578374    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.581536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.581515    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.632382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.632308    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.632519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.632424    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbw7\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\") pod \"e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") " pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:41.709269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:41.707496    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:42.055157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.055106    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"]
Nov 03 21:18:42.058542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.058513    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f44806f0-nctt5"]
Nov 03 21:18:42.139970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.139940    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:18:42.140673 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:18:42.140613    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc319af1b_55d4_4175_8425_fdb04cae767f.slice/crio-a4304c8368ef058fb2ff2a77f6a08d71fe4aa6e97927940a986a11684f78af1e WatchSource:0}: Error finding container a4304c8368ef058fb2ff2a77f6a08d71fe4aa6e97927940a986a11684f78af1e: Status 404 returned error can't find the container with id a4304c8368ef058fb2ff2a77f6a08d71fe4aa6e97927940a986a11684f78af1e
Nov 03 21:18:42.143358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.143321    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:42.143456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.143392    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:42.143456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.143434    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:18:42.364728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.364626    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerStarted","Data":"79924a3324e8395e18564fef1c7ef1d6a6dd4dd825977c7135d334f5491bc9f3"}
Nov 03 21:18:42.364728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.364702    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerStarted","Data":"a4304c8368ef058fb2ff2a77f6a08d71fe4aa6e97927940a986a11684f78af1e"}
Nov 03 21:18:42.482513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.482449    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podStartSLOduration=1.482429182 podStartE2EDuration="1.482429182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:42.479013062 +0000 UTC m=+1824.110326561" watchObservedRunningTime="2026-11-03 21:18:42.482429182 +0000 UTC m=+1824.113742662"
Nov 03 21:18:42.495037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.495003    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41be67c-0b5c-43f1-ac30-3e1e51d817b3" path="/var/lib/kubelet/pods/c41be67c-0b5c-43f1-ac30-3e1e51d817b3/volumes"
Nov 03 21:18:42.710404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:42.710364    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:43.715131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:43.715091    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:44.266851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:44.266821    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:18:44.267061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:44.266899    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:18:44.379543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:44.379511    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:44.380613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:44.380588    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:18:49.267865 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:18:49.267815    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:18:56.069725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:56.069609    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"]
Nov 03 21:18:56.114674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:56.114630    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32f6e43c-qsrz9"]
Nov 03 21:18:56.541065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:56.532792    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d7444a-0adc-45eb-b392-906070183a9c" path="/var/lib/kubelet/pods/25d7444a-0adc-45eb-b392-906070183a9c/volumes"
Nov 03 21:18:59.082569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:18:59.082530    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:19:05.584481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.584424    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"]
Nov 03 21:19:05.590118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.590098    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.641405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.641382    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"]
Nov 03 21:19:05.684177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.684154    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:19:05.684294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.684238    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:19:05.685773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.685752    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778r9\" (UniqueName: \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.685834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.685801    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.685912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.685898    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.786849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.786823    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778r9\" (UniqueName: \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.786979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.786885    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.786979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.786929    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.787553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.787534    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.789137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.789112    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.845000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.844947    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-778r9\" (UniqueName: \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\") pod \"maas-api-revoke-sub-4842a9ae-r79kc\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:05.904674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:05.904629    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:06.278126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:06.278086    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"]
Nov 03 21:19:06.279084 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:19:06.279040    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf2a840_049c_46b8_af0f_e0a3a9ff2a04.slice/crio-2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79 WatchSource:0}: Error finding container 2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79: Status 404 returned error can't find the container with id 2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79
Nov 03 21:19:06.585570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:06.585543    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc" event={"ID":"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04","Type":"ContainerStarted","Data":"858a21881c759443a295636c5f80bf8c80a2f46d1717a9be84af04ec00d44aaf"}
Nov 03 21:19:06.585911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:06.585593    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc" event={"ID":"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04","Type":"ContainerStarted","Data":"2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79"}
Nov 03 21:19:06.696491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:06.696436    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc" podStartSLOduration=1.6964183149999998 podStartE2EDuration="1.696418315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:06.694081815 +0000 UTC m=+1848.325395292" watchObservedRunningTime="2026-11-03 21:19:06.696418315 +0000 UTC m=+1848.327731793"
Nov 03 21:19:07.242282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.242248    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:19:07.242498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.242470    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" containerID="cri-o://93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683" gracePeriod=30
Nov 03 21:19:07.288785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.288749    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:19:07.288962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.288942    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" containerID="cri-o://21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2" gracePeriod=30
Nov 03 21:19:07.595379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.595344    5533 generic.go:356] "Generic (PLEG): container finished" podID="0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" containerID="858a21881c759443a295636c5f80bf8c80a2f46d1717a9be84af04ec00d44aaf" exitCode=0
Nov 03 21:19:07.595905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:07.595701    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc" event={"ID":"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04","Type":"ContainerDied","Data":"858a21881c759443a295636c5f80bf8c80a2f46d1717a9be84af04ec00d44aaf"}
Nov 03 21:19:08.758334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.758298    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:08.924184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.924161    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\") pod \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") "
Nov 03 21:19:08.924329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.924231    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\" (UniqueName: \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\") pod \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") "
Nov 03 21:19:08.924372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.924337    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\") pod \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\" (UID: \"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04\") "
Nov 03 21:19:08.924566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.924546    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca" pod "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" (UID: "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:08.924864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.924844    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:08.926334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.926313    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9" pod "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" (UID: "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04"). InnerVolumeSpecName "kube-api-access-778r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:08.926418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:08.926320    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca" pod "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" (UID: "0cf2a840-049c-46b8-af0f-e0a3a9ff2a04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:19:09.025516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:09.025490    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-778r9\" (UniqueName: \"kubernetes.io/projected/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-kube-api-access-778r9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:09.025516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:09.025514    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:09.610944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:09.610917    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"
Nov 03 21:19:09.610944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:09.610935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc" event={"ID":"0cf2a840-049c-46b8-af0f-e0a3a9ff2a04","Type":"ContainerDied","Data":"2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79"}
Nov 03 21:19:09.611161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:09.610979    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2965509b92c8fdb24094443aef6c4607bfd5d4f49cbf7509b43e9749f2c68f79"
Nov 03 21:19:11.886861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:11.886462    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:19:11.886861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:11.886523    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:19:12.425434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.425398    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 21:19:12.425557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.425472    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 21:19:12.576534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.576506    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:19:12.623450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.623424    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:19:12.637150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.637118    5533 generic.go:356] "Generic (PLEG): container finished" podID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerID="21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2" exitCode=0
Nov 03 21:19:12.637340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.637217    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" event={"ID":"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c","Type":"ContainerDied","Data":"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"}
Nov 03 21:19:12.637340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.637266    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx" event={"ID":"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c","Type":"ContainerDied","Data":"db1f143fe3192d25052f5f1fc0508ec55a4545ab2096209fc5dc679430090a59"}
Nov 03 21:19:12.637340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.637293    5533 scope.go:122] "RemoveContainer" containerID="21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"
Nov 03 21:19:12.637579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.637436    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"
Nov 03 21:19:12.640162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.640137    5533 generic.go:356] "Generic (PLEG): container finished" podID="110672cc-7b89-4860-a963-d39620d3486a" containerID="93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683" exitCode=0
Nov 03 21:19:12.640270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.640254    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" event={"ID":"110672cc-7b89-4860-a963-d39620d3486a","Type":"ContainerDied","Data":"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"}
Nov 03 21:19:12.640321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.640293    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs" event={"ID":"110672cc-7b89-4860-a963-d39620d3486a","Type":"ContainerDied","Data":"214b231d30527cb5927831fb91ff73ad59419666f5acce180ea80d201547e35e"}
Nov 03 21:19:12.640373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.640361    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"
Nov 03 21:19:12.649680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.649657    5533 scope.go:122] "RemoveContainer" containerID="21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"
Nov 03 21:19:12.649957 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:19:12.649934    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2\": container with ID starting with 21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2 not found: ID does not exist" containerID="21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"
Nov 03 21:19:12.650035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.649966    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2"} err="failed to get container status \"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2\": rpc error: code = NotFound desc = could not find container \"21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2\": container with ID starting with 21d61ea037cae836b965e25e730dafb279b2185e872f5a8c9ee355b4c6902dd2 not found: ID does not exist"
Nov 03 21:19:12.650035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.649984    5533 scope.go:122] "RemoveContainer" containerID="93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"
Nov 03 21:19:12.671180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671149    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671248    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671307    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671382    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671416    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671460    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671496    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671541    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.671732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.671587    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\") pod \"110672cc-7b89-4860-a963-d39620d3486a\" (UID: \"110672cc-7b89-4860-a963-d39620d3486a\") "
Nov 03 21:19:12.672614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.672588    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.672795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.672771    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.674018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.674000    5533 scope.go:122] "RemoveContainer" containerID="93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"
Nov 03 21:19:12.674514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.674463    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.674973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.674698    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.674973 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:19:12.674810    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683\": container with ID starting with 93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683 not found: ID does not exist" containerID="93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"
Nov 03 21:19:12.674973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.674848    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683"} err="failed to get container status \"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683\": rpc error: code = NotFound desc = could not find container \"93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683\": container with ID starting with 93bc8781a059ca9d652c9973e91cff139957ea7c59fb43bb4a29d55900b96683 not found: ID does not exist"
Nov 03 21:19:12.674973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.674860    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:12.677029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.677004    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:19:12.677967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.677933    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.678065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.678003    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "kube-api-access-z7hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:12.679207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.679183    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token" pod "110672cc-7b89-4860-a963-d39620d3486a" (UID: "110672cc-7b89-4860-a963-d39620d3486a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:12.773149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773105    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773213    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773368    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.773445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773426    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.773530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773510    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773545    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773582    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773697    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773737    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773816    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.773883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773844    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.773883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.773866    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\") pod \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\" (UID: \"2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c\") "
Nov 03 21:19:12.774351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774307    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774359    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774381    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774396    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774409    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774422    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774439    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110672cc-7b89-4860-a963-d39620d3486a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774452    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774478    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774492    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hnf\" (UniqueName: \"kubernetes.io/projected/110672cc-7b89-4860-a963-d39620d3486a-kube-api-access-z7hnf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774506    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774519    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110672cc-7b89-4860-a963-d39620d3486a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774532    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110672cc-7b89-4860-a963-d39620d3486a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.774866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.774447    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:12.776599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.776566    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:19:12.776979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.776957    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:19:12.777236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.777213    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:12.779474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.779451    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf" pod "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" (UID: "2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c"). InnerVolumeSpecName "kube-api-access-hzcmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891827    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891866    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891884    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891902    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891917    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:12.893716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:12.891931    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcmf\" (UniqueName: \"kubernetes.io/projected/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c-kube-api-access-hzcmf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:13.061902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:13.061862    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:19:13.069450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:13.069423    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-m7nzs"]
Nov 03 21:19:13.170238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:13.170210    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:19:13.174804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:13.174773    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-4v7mx"]
Nov 03 21:19:14.495471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:14.495438    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110672cc-7b89-4860-a963-d39620d3486a" path="/var/lib/kubelet/pods/110672cc-7b89-4860-a963-d39620d3486a/volumes"
Nov 03 21:19:14.496013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:14.495994    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" path="/var/lib/kubelet/pods/2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c/volumes"
Nov 03 21:19:16.497994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:16.497962    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:19:16.605437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:16.605399    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:19:19.260766 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:19:19.260732    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:19:19.506360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:19.506331    5533 scope.go:122] "RemoveContainer" containerID="3fd32f2783ebe36278edbdcf50a6a4513dd855cb7dbd3d98cb4239a4bd509fb0"
Nov 03 21:19:19.515805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:19.515783    5533 scope.go:122] "RemoveContainer" containerID="50b95816784d2112efa22b7c237bc913b74e756a9f826d8a331fcd33044959a4"
Nov 03 21:19:19.524718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:19.524698    5533 scope.go:122] "RemoveContainer" containerID="fb87c3f21e13243a01467f84097facf05a80b8c4f778fdf57a6babd9dd7a29fa"
Nov 03 21:19:21.498446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.498406    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerName="payload-processing" containerID="cri-o://c79da4ba0eb710d4c9e6ab495b9754423a53fafbb9577f252874efca5b360462" gracePeriod=25
Nov 03 21:19:21.606291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.606250    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" podUID="86aabb67-b543-4907-892f-c5196eade0c4" containerName="payload-pre-processing" containerID="cri-o://bbc1f2ff7f4959139d144498a02a9d2c88273290394a324cadcfecd2fc099ec5" gracePeriod=25
Nov 03 21:19:21.609126 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:19:21.609108    5533 logging.go:55] [core] [Channel #3141 SubChannel #3143]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.146:50052", ServerName: "10.133.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.146:50052: connect: connection refused"
Nov 03 21:19:21.609227 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:19:21.609143    5533 logging.go:55] [core] [Channel #3140 SubChannel #3142]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.145:50052", ServerName: "10.133.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.145:50052: connect: connection refused"
Nov 03 21:19:21.772200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.772169    5533 generic.go:356] "Generic (PLEG): container finished" podID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerID="c79da4ba0eb710d4c9e6ab495b9754423a53fafbb9577f252874efca5b360462" exitCode=0
Nov 03 21:19:21.772337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.772258    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" event={"ID":"8aa9b36b-9b0c-4562-942f-a425c4665edd","Type":"ContainerDied","Data":"c79da4ba0eb710d4c9e6ab495b9754423a53fafbb9577f252874efca5b360462"}
Nov 03 21:19:21.772337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.772323    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" event={"ID":"8aa9b36b-9b0c-4562-942f-a425c4665edd","Type":"ContainerDied","Data":"c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd"}
Nov 03 21:19:21.772436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.772345    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d1ce9d3d99243734fa0c37c09917d1a5ac160c9e1e7aa9baab137b0178edfd"
Nov 03 21:19:21.774387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.774364    5533 generic.go:356] "Generic (PLEG): container finished" podID="86aabb67-b543-4907-892f-c5196eade0c4" containerID="bbc1f2ff7f4959139d144498a02a9d2c88273290394a324cadcfecd2fc099ec5" exitCode=0
Nov 03 21:19:21.774487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.774423    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" event={"ID":"86aabb67-b543-4907-892f-c5196eade0c4","Type":"ContainerDied","Data":"bbc1f2ff7f4959139d144498a02a9d2c88273290394a324cadcfecd2fc099ec5"}
Nov 03 21:19:21.784417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.784396    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:19:21.894922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.894769    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:19:21.912338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.912310    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\") pod \"8aa9b36b-9b0c-4562-942f-a425c4665edd\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") "
Nov 03 21:19:21.912471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.912454    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\" (UniqueName: \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\") pod \"8aa9b36b-9b0c-4562-942f-a425c4665edd\" (UID: \"8aa9b36b-9b0c-4562-942f-a425c4665edd\") "
Nov 03 21:19:21.912729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.912705    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume" pod "8aa9b36b-9b0c-4562-942f-a425c4665edd" (UID: "8aa9b36b-9b0c-4562-942f-a425c4665edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:21.913140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.913088    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9b36b-9b0c-4562-942f-a425c4665edd-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:21.914591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:21.914564    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl" pod "8aa9b36b-9b0c-4562-942f-a425c4665edd" (UID: "8aa9b36b-9b0c-4562-942f-a425c4665edd"). InnerVolumeSpecName "kube-api-access-7n2xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:22.014234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.014123    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\" (UniqueName: \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\") pod \"86aabb67-b543-4907-892f-c5196eade0c4\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") "
Nov 03 21:19:22.014362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.014271    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\") pod \"86aabb67-b543-4907-892f-c5196eade0c4\" (UID: \"86aabb67-b543-4907-892f-c5196eade0c4\") "
Nov 03 21:19:22.014869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.014849    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2xl\" (UniqueName: \"kubernetes.io/projected/8aa9b36b-9b0c-4562-942f-a425c4665edd-kube-api-access-7n2xl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:22.014937 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.014905    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume" pod "86aabb67-b543-4907-892f-c5196eade0c4" (UID: "86aabb67-b543-4907-892f-c5196eade0c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:19:22.017095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.017067    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws" pod "86aabb67-b543-4907-892f-c5196eade0c4" (UID: "86aabb67-b543-4907-892f-c5196eade0c4"). InnerVolumeSpecName "kube-api-access-bs8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:19:22.116313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.116287    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8ws\" (UniqueName: \"kubernetes.io/projected/86aabb67-b543-4907-892f-c5196eade0c4-kube-api-access-bs8ws\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:22.116313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.116310    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aabb67-b543-4907-892f-c5196eade0c4-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:19:22.609488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.609447    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.145:50052\" within 1s: context deadline exceeded"
Nov 03 21:19:22.609854 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:19:22.609534    5533 logging.go:55] [core] [Channel #3140 SubChannel #3142]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.145:50052", ServerName: "10.133.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.145:50052: operation was canceled"
Nov 03 21:19:22.609854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.609534    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" podUID="86aabb67-b543-4907-892f-c5196eade0c4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.146:50052\" within 1s: context deadline exceeded"
Nov 03 21:19:22.609854 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:19:22.609602    5533 logging.go:55] [core] [Channel #3141 SubChannel #3143]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.146:50052", ServerName: "10.133.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.146:50052: operation was canceled"
Nov 03 21:19:22.783004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.782935    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"
Nov 03 21:19:22.783004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.782935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf" event={"ID":"86aabb67-b543-4907-892f-c5196eade0c4","Type":"ContainerDied","Data":"7e03f64392b292e39e2e8b7940336387ae9034e5f3964397dd0e49a88ca8981a"}
Nov 03 21:19:22.783225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.783040    5533 scope.go:122] "RemoveContainer" containerID="bbc1f2ff7f4959139d144498a02a9d2c88273290394a324cadcfecd2fc099ec5"
Nov 03 21:19:22.783355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.783284    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"
Nov 03 21:19:22.942910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.942878    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:19:22.950167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:22.950140    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eae138-c5f8bdcb6-pd9tx"]
Nov 03 21:19:23.053177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:23.053089    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:19:23.057345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:23.057318    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eae138-645fbcdccf-qrmrf"]
Nov 03 21:19:24.495911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:24.495869    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86aabb67-b543-4907-892f-c5196eade0c4" path="/var/lib/kubelet/pods/86aabb67-b543-4907-892f-c5196eade0c4/volumes"
Nov 03 21:19:24.496442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:24.496421    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" path="/var/lib/kubelet/pods/8aa9b36b-9b0c-4562-942f-a425c4665edd/volumes"
Nov 03 21:19:38.271893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:38.271835    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:19:49.052060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.052024    5533 generic.go:356] "Generic (PLEG): container finished" podID="44557d39-db06-4b1c-b42a-8168134023b5" containerID="cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330" exitCode=0
Nov 03 21:19:49.052547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.052057    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerDied","Data":"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"}
Nov 03 21:19:49.052615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.052559    5533 scope.go:122] "RemoveContainer" containerID="cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"
Nov 03 21:19:49.054425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.054388    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:19:49.054552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.054476    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:19:49.054552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:49.054521    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:19:49.276625 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:19:49.276110    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:19:50.073673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:50.071606    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerStarted","Data":"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"}
Nov 03 21:19:50.105324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:50.105290    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:19:50.105555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:50.105531    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:50.105605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:50.105589    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:51.106057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:51.106016    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:51.106520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:51.106073    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:52.105141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:52.105086    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:52.105335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:52.105171    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:53.106019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:53.105979    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:53.106424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:53.106079    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:54.105867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:54.105834    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:54.106149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:54.106117    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:55.105140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:55.105090    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 21:19:55.105368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:55.105168    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 21:19:56.109802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:56.109774    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:19:56.110218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:56.110034    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:19:56.110681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:19:56.110659    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:20:05.816070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:05.815989    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:20:15.181044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.180728    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:37566->10.133.0.152:15021: read: connection reset by peer" start-of-body=
Nov 03 21:20:15.181044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.180785    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:37566->10.133.0.152:15021: read: connection reset by peer"
Nov 03 21:20:15.257502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.257472    5533 generic.go:356] "Generic (PLEG): container finished" podID="c319af1b-55d4-4175-8425-fdb04cae767f" containerID="79924a3324e8395e18564fef1c7ef1d6a6dd4dd825977c7135d334f5491bc9f3" exitCode=0
Nov 03 21:20:15.257614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.257545    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerDied","Data":"79924a3324e8395e18564fef1c7ef1d6a6dd4dd825977c7135d334f5491bc9f3"}
Nov 03 21:20:15.257949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.257927    5533 scope.go:122] "RemoveContainer" containerID="79924a3324e8395e18564fef1c7ef1d6a6dd4dd825977c7135d334f5491bc9f3"
Nov 03 21:20:15.259333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.259303    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:20:15.259405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.259367    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:20:15.259405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:15.259397    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:20:16.283257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:16.283222    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerStarted","Data":"72620d9489c5ec37b22c34c01738cb8a124c6e94930d0a6a1656a3a04869a0e2"}
Nov 03 21:20:16.710611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:16.710514    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:20:16.711202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:16.711107    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:20:16.711772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:16.711619    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 21:20:17.710193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:17.710152    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:20:17.710711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:17.710212    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 21:20:18.710797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:18.710761    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:20:18.711177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:18.710813    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 21:20:19.294574 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:20:19.294534    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:20:19.581726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:19.579261    5533 scope.go:122] "RemoveContainer" containerID="33d363c8892ed85f4d135c8a9b241133f2236f0aee24645a383286ec52f04c19"
Nov 03 21:20:19.710768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:19.710721    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:20:19.710912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:19.710797    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 21:20:20.709990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:20.709953    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 21:20:20.710166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:20.710003    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 21:20:21.714538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:21.714508    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:20:21.715029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:21.714961    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:20:21.715627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:21.715606    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:20:36.128388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:36.128349    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:20:36.254951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:36.254914    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:20:39.401464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:39.401424    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:20:41.129741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.129689    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" podUID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" containerName="payload-processing" containerID="cri-o://da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8" gracePeriod=25
Nov 03 21:20:41.255729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.255671    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" podUID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" containerName="payload-pre-processing" containerID="cri-o://e597fca9b34fb6287b7089211c0278d37398704b80ae279054cd127cbe7132db" gracePeriod=25
Nov 03 21:20:41.395007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.394948    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:20:41.464295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.464107    5533 generic.go:356] "Generic (PLEG): container finished" podID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" containerID="e597fca9b34fb6287b7089211c0278d37398704b80ae279054cd127cbe7132db" exitCode=0
Nov 03 21:20:41.464295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.464215    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" event={"ID":"0e72669b-fc4e-412a-aa3c-eccc975ec9c4","Type":"ContainerDied","Data":"e597fca9b34fb6287b7089211c0278d37398704b80ae279054cd127cbe7132db"}
Nov 03 21:20:41.466581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.466552    5533 generic.go:356] "Generic (PLEG): container finished" podID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" containerID="da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8" exitCode=0
Nov 03 21:20:41.466809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.466665    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"
Nov 03 21:20:41.466958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.466669    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\" (UniqueName: \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\") pod \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") "
Nov 03 21:20:41.467042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.467025    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\") pod \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\" (UID: \"a79ce08f-f101-44c9-ab8e-597c8b5e4adf\") "
Nov 03 21:20:41.467121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.466689    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" event={"ID":"a79ce08f-f101-44c9-ab8e-597c8b5e4adf","Type":"ContainerDied","Data":"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"}
Nov 03 21:20:41.467184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.467128    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8" event={"ID":"a79ce08f-f101-44c9-ab8e-597c8b5e4adf","Type":"ContainerDied","Data":"9f241907078456bf027cf76cf3f4cffc79bd12adad9ebb5411410bbd0d4e4794"}
Nov 03 21:20:41.467184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.467165    5533 scope.go:122] "RemoveContainer" containerID="da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"
Nov 03 21:20:41.467817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.467793    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume" pod "a79ce08f-f101-44c9-ab8e-597c8b5e4adf" (UID: "a79ce08f-f101-44c9-ab8e-597c8b5e4adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:41.471312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.471265    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s" pod "a79ce08f-f101-44c9-ab8e-597c8b5e4adf" (UID: "a79ce08f-f101-44c9-ab8e-597c8b5e4adf"). InnerVolumeSpecName "kube-api-access-ff56s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:41.493127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.493104    5533 scope.go:122] "RemoveContainer" containerID="da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"
Nov 03 21:20:41.493427 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:20:41.493392    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8\": container with ID starting with da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8 not found: ID does not exist" containerID="da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"
Nov 03 21:20:41.493530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.493424    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8"} err="failed to get container status \"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8\": rpc error: code = NotFound desc = could not find container \"da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8\": container with ID starting with da45c90f3b3a743fe0759f06835d1fd84cd2bc0cea8a76d858a5f0504ac9aba8 not found: ID does not exist"
Nov 03 21:20:41.516320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.516301    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:20:41.568884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.568860    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff56s\" (UniqueName: \"kubernetes.io/projected/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-kube-api-access-ff56s\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:41.568884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.568884    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79ce08f-f101-44c9-ab8e-597c8b5e4adf-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:41.669497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.669444    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\") pod \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") "
Nov 03 21:20:41.669727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.669533    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\" (UniqueName: \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\") pod \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\" (UID: \"0e72669b-fc4e-412a-aa3c-eccc975ec9c4\") "
Nov 03 21:20:41.669889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.669867    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume" pod "0e72669b-fc4e-412a-aa3c-eccc975ec9c4" (UID: "0e72669b-fc4e-412a-aa3c-eccc975ec9c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:41.670033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.670016    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:41.671487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.671469    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw" pod "0e72669b-fc4e-412a-aa3c-eccc975ec9c4" (UID: "0e72669b-fc4e-412a-aa3c-eccc975ec9c4"). InnerVolumeSpecName "kube-api-access-zjnmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:41.770867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.770840    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnmw\" (UniqueName: \"kubernetes.io/projected/0e72669b-fc4e-412a-aa3c-eccc975ec9c4-kube-api-access-zjnmw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:41.882043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.882012    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:20:41.887803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:41.887774    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-6d99b9ff64-v5pw8"]
Nov 03 21:20:42.474791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.474754    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp" event={"ID":"0e72669b-fc4e-412a-aa3c-eccc975ec9c4","Type":"ContainerDied","Data":"a96f11d5bfa91aeb254daa164ed4d13311caa9b71feea6194b03f9e5af313239"}
Nov 03 21:20:42.475240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.474759    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"
Nov 03 21:20:42.475240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.474850    5533 scope.go:122] "RemoveContainer" containerID="e597fca9b34fb6287b7089211c0278d37398704b80ae279054cd127cbe7132db"
Nov 03 21:20:42.494086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.494063    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" path="/var/lib/kubelet/pods/a79ce08f-f101-44c9-ab8e-597c8b5e4adf/volumes"
Nov 03 21:20:42.630586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.630565    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:20:42.677106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:42.677084    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-812ef9-6ff67d5959-2mwlp"]
Nov 03 21:20:44.493035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:44.493004    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" path="/var/lib/kubelet/pods/0e72669b-fc4e-412a-aa3c-eccc975ec9c4/volumes"
Nov 03 21:20:47.398044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398013    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"]
Nov 03 21:20:47.398527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398512    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398529    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398548    5533 cpu_manager.go:414] "containerMap: removing container" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398554    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398569    5533 cpu_manager.go:414] "containerMap: removing container" podUID="86aabb67-b543-4907-892f-c5196eade0c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398574    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86aabb67-b543-4907-892f-c5196eade0c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398581    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" containerName="revoke-keys"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398586    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" containerName="revoke-keys"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398595    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398600    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398616    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398621    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398630    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398661    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398750    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="86aabb67-b543-4907-892f-c5196eade0c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398762    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" containerName="revoke-keys"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398770    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa9b36b-9b0c-4562-942f-a425c4665edd" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398778    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e72669b-fc4e-412a-aa3c-eccc975ec9c4" containerName="payload-pre-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398786    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79ce08f-f101-44c9-ab8e-597c8b5e4adf" containerName="payload-processing"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398793    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6fa1ac-1067-4c19-8fb6-d3c76493cd1c" containerName="istio-proxy"
Nov 03 21:20:47.398826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.398802    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="110672cc-7b89-4860-a963-d39620d3486a" containerName="istio-proxy"
Nov 03 21:20:47.403376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.403356    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.457131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.457101    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"]
Nov 03 21:20:47.499826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.499800    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:20:47.499826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.499814    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:20:47.531899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.531867    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbmd\" (UniqueName: \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.532029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.531917    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.532189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.532163    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.632814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.632782    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbmd\" (UniqueName: \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.632979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.632835    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.633383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.633356    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.633611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.633584    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.635827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.635801    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.690754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.690729    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbmd\" (UniqueName: \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\") pod \"maas-api-revoke-sub-8703dc76-plhcv\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:47.720987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:47.720962    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:48.092283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:48.092253    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"]
Nov 03 21:20:48.093195 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:20:48.093167    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a550e1e_d442_478d_b1f6_7275066b2067.slice/crio-9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0 WatchSource:0}: Error finding container 9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0: Status 404 returned error can't find the container with id 9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0
Nov 03 21:20:48.522976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:48.522943    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv" event={"ID":"6a550e1e-d442-478d-b1f6-7275066b2067","Type":"ContainerStarted","Data":"2dddee8cdb02bd944e21dda953bf7d5a7c53f1f2a8b721496a4cd04dd74a44b4"}
Nov 03 21:20:48.523371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:48.522994    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv" event={"ID":"6a550e1e-d442-478d-b1f6-7275066b2067","Type":"ContainerStarted","Data":"9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0"}
Nov 03 21:20:48.635010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:48.634902    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv" podStartSLOduration=1.634882348 podStartE2EDuration="1.634882348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:48.632143108 +0000 UTC m=+1950.263456584" watchObservedRunningTime="2026-11-03 21:20:48.634882348 +0000 UTC m=+1950.266195828"
Nov 03 21:20:49.269154 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:20:49.269066    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:20:49.627746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:49.627630    5533 generic.go:356] "Generic (PLEG): container finished" podID="6a550e1e-d442-478d-b1f6-7275066b2067" containerID="2dddee8cdb02bd944e21dda953bf7d5a7c53f1f2a8b721496a4cd04dd74a44b4" exitCode=0
Nov 03 21:20:49.628240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:49.627907    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv" event={"ID":"6a550e1e-d442-478d-b1f6-7275066b2067","Type":"ContainerDied","Data":"2dddee8cdb02bd944e21dda953bf7d5a7c53f1f2a8b721496a4cd04dd74a44b4"}
Nov 03 21:20:50.795127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.795098    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:50.803022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.802995    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\") pod \"6a550e1e-d442-478d-b1f6-7275066b2067\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") "
Nov 03 21:20:50.803157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.803030    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\" (UniqueName: \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\") pod \"6a550e1e-d442-478d-b1f6-7275066b2067\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") "
Nov 03 21:20:50.803157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.803090    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\") pod \"6a550e1e-d442-478d-b1f6-7275066b2067\" (UID: \"6a550e1e-d442-478d-b1f6-7275066b2067\") "
Nov 03 21:20:50.803391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.803365    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca" pod "6a550e1e-d442-478d-b1f6-7275066b2067" (UID: "6a550e1e-d442-478d-b1f6-7275066b2067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:50.803562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.803536    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:50.805038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.805013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca" pod "6a550e1e-d442-478d-b1f6-7275066b2067" (UID: "6a550e1e-d442-478d-b1f6-7275066b2067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:50.805149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.805129    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd" pod "6a550e1e-d442-478d-b1f6-7275066b2067" (UID: "6a550e1e-d442-478d-b1f6-7275066b2067"). InnerVolumeSpecName "kube-api-access-jrbmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:50.904211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.904155    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrbmd\" (UniqueName: \"kubernetes.io/projected/6a550e1e-d442-478d-b1f6-7275066b2067-kube-api-access-jrbmd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:50.904211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:50.904180    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a550e1e-d442-478d-b1f6-7275066b2067-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:51.642110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:51.642065    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv" event={"ID":"6a550e1e-d442-478d-b1f6-7275066b2067","Type":"ContainerDied","Data":"9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0"}
Nov 03 21:20:51.642110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:51.642094    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"
Nov 03 21:20:51.642344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:51.642119    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bcb51f53c7f028b052df6c0a90ee7557f3ff1c24ce6e8ebf4f4bf0c7e569ce0"
Nov 03 21:20:53.462709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:53.462673    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:20:53.463101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:53.462875    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" containerID="cri-o://261f0ba8a8ee549ec3ae5e9e0c7d4958d1f426803e1d21dc36783d5f7bb65d77" gracePeriod=30
Nov 03 21:20:58.702285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.702248    5533 generic.go:356] "Generic (PLEG): container finished" podID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerID="261f0ba8a8ee549ec3ae5e9e0c7d4958d1f426803e1d21dc36783d5f7bb65d77" exitCode=0
Nov 03 21:20:58.702728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.702321    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerDied","Data":"261f0ba8a8ee549ec3ae5e9e0c7d4958d1f426803e1d21dc36783d5f7bb65d77"}
Nov 03 21:20:58.702728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.702410    5533 scope.go:122] "RemoveContainer" containerID="db6b62367bf7c46e591b73bfb20ea61e4181ac814aa5274acbcc810967498649"
Nov 03 21:20:58.822697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.822673    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:20:58.915189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915160    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915199    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915269    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915332    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915588    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:58.915737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915615    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915685    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915728    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915762    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.915926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.915820    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\") pod \"1a72afed-8dab-4432-8ab3-f1fde1647085\" (UID: \"1a72afed-8dab-4432-8ab3-f1fde1647085\") "
Nov 03 21:20:58.917170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.917139    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:58.920830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.919560    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:58.920830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.919627    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:58.920830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.919701    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:58.920830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.919804    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:58.922277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.922227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:58.923489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.923462    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:20:58.941772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.928090    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "kube-api-access-kh5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:58.941772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:58.935934    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert" pod "1a72afed-8dab-4432-8ab3-f1fde1647085" (UID: "1a72afed-8dab-4432-8ab3-f1fde1647085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:59.018077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018037    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a72afed-8dab-4432-8ab3-f1fde1647085-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018070    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018082    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh5h4\" (UniqueName: \"kubernetes.io/projected/1a72afed-8dab-4432-8ab3-f1fde1647085-kube-api-access-kh5h4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018093    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018102    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018110    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018119    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.018293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.018127    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a72afed-8dab-4432-8ab3-f1fde1647085-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:59.158345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.158262    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:20:59.158864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.158846    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.158923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.158866    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.158923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.158880    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6a550e1e-d442-478d-b1f6-7275066b2067" containerName="revoke-keys"
Nov 03 21:20:59.158923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.158885    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a550e1e-d442-478d-b1f6-7275066b2067" containerName="revoke-keys"
Nov 03 21:20:59.159020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.159009    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.159020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.159020    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a550e1e-d442-478d-b1f6-7275066b2067" containerName="revoke-keys"
Nov 03 21:20:59.162865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.162842    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.220658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.220060    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7af9dd\""
Nov 03 21:20:59.224653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.221315    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49tq\" (UniqueName: \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.224653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.221457    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.224653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.222437    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7af9dd-dockercfg-9bdbk\""
Nov 03 21:20:59.248656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.245224    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:20:59.323152 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.323107    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.323329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.323238    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49tq\" (UniqueName: \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.323892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.323864    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.340809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.340781    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:20:59.341283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.341270    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.341343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.341284    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.341401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.341392    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy"
Nov 03 21:20:59.344882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.344866    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.381081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.381054    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49tq\" (UniqueName: \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\") pod \"payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.399430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.399403    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:20:59.424470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.424442    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.424615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.424594    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfnj\" (UniqueName: \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.487917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.487894    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:20:59.526335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.526305    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfnj\" (UniqueName: \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.526476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.526413    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.527241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.527213    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.584155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.584114    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfnj\" (UniqueName: \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\") pod \"payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.637168 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.637137    5533 patch_prober.go:29] interesting pod/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:20:59.637317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.637193    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:20:59.658130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.658106    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:20:59.709468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.709391    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q" event={"ID":"1a72afed-8dab-4432-8ab3-f1fde1647085","Type":"ContainerDied","Data":"605e135464d3a48de21791fa221887d0a920e99dfb0e814e7de133ff13caf2e8"}
Nov 03 21:20:59.709468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.709416    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"
Nov 03 21:20:59.709946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.709492    5533 scope.go:122] "RemoveContainer" containerID="261f0ba8a8ee549ec3ae5e9e0c7d4958d1f426803e1d21dc36783d5f7bb65d77"
Nov 03 21:20:59.855014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.854990    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:20:59.855780 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:20:59.855754    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e604671_7acb_4eb6_bc95_2ff428cb3c13.slice/crio-46d0bbc8e6fd9d4ca8038d62589307d4af063eee7f426caad613b957bbadb512 WatchSource:0}: Error finding container 46d0bbc8e6fd9d4ca8038d62589307d4af063eee7f426caad613b957bbadb512: Status 404 returned error can't find the container with id 46d0bbc8e6fd9d4ca8038d62589307d4af063eee7f426caad613b957bbadb512
Nov 03 21:20:59.866876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.866851    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:20:59.873297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:20:59.873276    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2ad2d-openshift-default-b64c5577f-m648q"]
Nov 03 21:21:00.030887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.030860    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:21:00.032172 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:21:00.032148    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8f1b41_9509_44c7_8011_5b178e219613.slice/crio-d10fae28f83a6c981889f093c18befe6fa5baa75cabd4da259e7eae567a98637 WatchSource:0}: Error finding container d10fae28f83a6c981889f093c18befe6fa5baa75cabd4da259e7eae567a98637: Status 404 returned error can't find the container with id d10fae28f83a6c981889f093c18befe6fa5baa75cabd4da259e7eae567a98637
Nov 03 21:21:00.493886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.493858    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a72afed-8dab-4432-8ab3-f1fde1647085" path="/var/lib/kubelet/pods/1a72afed-8dab-4432-8ab3-f1fde1647085/volumes"
Nov 03 21:21:00.717350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.717307    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" event={"ID":"cb8f1b41-9509-44c7-8011-5b178e219613","Type":"ContainerStarted","Data":"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"}
Nov 03 21:21:00.717809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.717381    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:21:00.717809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.717408    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" event={"ID":"cb8f1b41-9509-44c7-8011-5b178e219613","Type":"ContainerStarted","Data":"d10fae28f83a6c981889f093c18befe6fa5baa75cabd4da259e7eae567a98637"}
Nov 03 21:21:00.720076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.720050    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" event={"ID":"1e604671-7acb-4eb6-bc95-2ff428cb3c13","Type":"ContainerStarted","Data":"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"}
Nov 03 21:21:00.720213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.720095    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" event={"ID":"1e604671-7acb-4eb6-bc95-2ff428cb3c13","Type":"ContainerStarted","Data":"46d0bbc8e6fd9d4ca8038d62589307d4af063eee7f426caad613b957bbadb512"}
Nov 03 21:21:00.720213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.720119    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:21:00.826840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.826750    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" podStartSLOduration=1.8267359330000001 podStartE2EDuration="1.826735933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:00.82616027 +0000 UTC m=+1962.457473745" watchObservedRunningTime="2026-11-03 21:21:00.826735933 +0000 UTC m=+1962.458049409"
Nov 03 21:21:00.935348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:00.935296    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" podStartSLOduration=1.935280181 podStartE2EDuration="1.935280181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:00.935018384 +0000 UTC m=+1962.566331861" watchObservedRunningTime="2026-11-03 21:21:00.935280181 +0000 UTC m=+1962.566593658"
Nov 03 21:21:04.691249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.691213    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:21:04.695815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.695790    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.770658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.755828    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:21:04.802671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.793513    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.802671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.793811    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm96l\" (UniqueName: \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.802671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.794187    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1190cc\""
Nov 03 21:21:04.802671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.794281    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1190cc-dockercfg-vcf5p\""
Nov 03 21:21:04.863687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.863649    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:21:04.868186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.868162    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:04.894497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.894467    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm96l\" (UniqueName: \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.894659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.894557    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.895223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.895204    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.923906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.923879    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:21:04.952983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.952914    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm96l\" (UniqueName: \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\") pod \"payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:04.995731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.995701    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5gb\" (UniqueName: \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:04.995857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:04.995833    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.016488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.016462    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:05.097352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.096676    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.097352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.096900    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5gb\" (UniqueName: \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.098216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.098184    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.157186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.157153    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5gb\" (UniqueName: \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\") pod \"payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.182162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.182138    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.386264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.386219    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:21:05.387481 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:21:05.387434    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cae954_b716_4b94_afda_305d058fec58.slice/crio-c1e2d94807a1b5047922a258f495a486391fea9080609f91783e1dce370d1719 WatchSource:0}: Error finding container c1e2d94807a1b5047922a258f495a486391fea9080609f91783e1dce370d1719: Status 404 returned error can't find the container with id c1e2d94807a1b5047922a258f495a486391fea9080609f91783e1dce370d1719
Nov 03 21:21:05.553558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.553533    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:21:05.553733 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:21:05.553704    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3924f959_c529_46ae_ad5b_09dae3d93eb4.slice/crio-c17dc9fa2853c7628ffb1f75633b69de0f0bd0371d0c51334509bb3c6103a96e WatchSource:0}: Error finding container c17dc9fa2853c7628ffb1f75633b69de0f0bd0371d0c51334509bb3c6103a96e: Status 404 returned error can't find the container with id c17dc9fa2853c7628ffb1f75633b69de0f0bd0371d0c51334509bb3c6103a96e
Nov 03 21:21:05.759869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.759837    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" event={"ID":"3924f959-c529-46ae-ad5b-09dae3d93eb4","Type":"ContainerStarted","Data":"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"}
Nov 03 21:21:05.760287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.759898    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" event={"ID":"3924f959-c529-46ae-ad5b-09dae3d93eb4","Type":"ContainerStarted","Data":"c17dc9fa2853c7628ffb1f75633b69de0f0bd0371d0c51334509bb3c6103a96e"}
Nov 03 21:21:05.760287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.759932    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:05.761791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.761765    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" event={"ID":"b7cae954-b716-4b94-afda-305d058fec58","Type":"ContainerStarted","Data":"6af2675aac0a1ed059a94917e9ba4fa025a36fbb1620dacf5d9f77521561de4c"}
Nov 03 21:21:05.761898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.761815    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" event={"ID":"b7cae954-b716-4b94-afda-305d058fec58","Type":"ContainerStarted","Data":"c1e2d94807a1b5047922a258f495a486391fea9080609f91783e1dce370d1719"}
Nov 03 21:21:05.761898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.761870    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:05.872478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.872391    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" podStartSLOduration=1.8723776170000002 podStartE2EDuration="1.872377617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:05.869003613 +0000 UTC m=+1967.500317093" watchObservedRunningTime="2026-11-03 21:21:05.872377617 +0000 UTC m=+1967.503691094"
Nov 03 21:21:05.978106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:05.978055    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" podStartSLOduration=1.9780378299999999 podStartE2EDuration="1.97803783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:05.977832359 +0000 UTC m=+1967.609145835" watchObservedRunningTime="2026-11-03 21:21:05.97803783 +0000 UTC m=+1967.609351308"
Nov 03 21:21:11.730271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:11.730239    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:21:11.730677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:11.730306    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:21:16.772311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:16.772281    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:21:16.772811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:16.772352    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:21:19.262145 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:21:19.262109    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:21:24.451993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:24.451957    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:21:29.102596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.102564    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:21:29.108526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.108502    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.155084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.155056    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.155194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.155098    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzn9v\" (UniqueName: \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.163273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.163200    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:21:29.204953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.204922    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-812ef9\""
Nov 03 21:21:29.205083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.204981    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-812ef9-dockercfg-tvxcf\""
Nov 03 21:21:29.255932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.255909    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.256055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.255946    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzn9v\" (UniqueName: \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.256619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.256598    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.316080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.316052    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzn9v\" (UniqueName: \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\") pod \"payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.422789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.422761    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:29.798997 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:21:29.798959    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296697e9_2fc1_49a0_a004_5176cf8d125a.slice/crio-082b3256b127b26cfb384e05e9cf90db6664efe70600822e915c41fc8da7e73e WatchSource:0}: Error finding container 082b3256b127b26cfb384e05e9cf90db6664efe70600822e915c41fc8da7e73e: Status 404 returned error can't find the container with id 082b3256b127b26cfb384e05e9cf90db6664efe70600822e915c41fc8da7e73e
Nov 03 21:21:29.799811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:29.799785    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:21:30.017282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:30.017245    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" event={"ID":"296697e9-2fc1-49a0-a004-5176cf8d125a","Type":"ContainerStarted","Data":"0d92f2266e4089b2f05f7b1bdf219d1c30af759e7895e2682c1be63c1bc21a6d"}
Nov 03 21:21:30.017485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:30.017302    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" event={"ID":"296697e9-2fc1-49a0-a004-5176cf8d125a","Type":"ContainerStarted","Data":"082b3256b127b26cfb384e05e9cf90db6664efe70600822e915c41fc8da7e73e"}
Nov 03 21:21:30.017485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:30.017370    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:30.142075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:30.138165    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" podStartSLOduration=1.138144133 podStartE2EDuration="1.138144133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:30.128462 +0000 UTC m=+1991.759775478" watchObservedRunningTime="2026-11-03 21:21:30.138144133 +0000 UTC m=+1991.769457611"
Nov 03 21:21:41.025592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:21:41.025557    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:21:49.263184 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:21:49.263145    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:22:13.103382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:13.103346    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:22:19.261375 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:22:19.261328    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:22:23.798765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:23.798727    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:22:23.799346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:23.798925    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" containerID="cri-o://f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96" gracePeriod=30
Nov 03 21:22:26.131234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:26.131195    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:22:26.131691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:26.131254    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:22:29.140703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.140677    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:22:29.256610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256579    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256618    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256682    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256708    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256801    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256851    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256873    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256901    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.256954    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\") pod \"9519b47f-00b3-4316-8620-0cf0b64af21d\" (UID: \"9519b47f-00b3-4316-8620-0cf0b64af21d\") "
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257035    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257128    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:29.257349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257133    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:29.258010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257420    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:29.258010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257502    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9519b47f-00b3-4316-8620-0cf0b64af21d-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.258010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257524    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.258010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257537    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.258010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.257503    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:29.259116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.259059    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:29.259238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.259168    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:29.259238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.259179    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "kube-api-access-lvvm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:29.259344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.259264    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token" pod "9519b47f-00b3-4316-8620-0cf0b64af21d" (UID: "9519b47f-00b3-4316-8620-0cf0b64af21d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358421    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvm6\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-kube-api-access-lvvm6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358449    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358460    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358470    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358479    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.358490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.358488    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9519b47f-00b3-4316-8620-0cf0b64af21d-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:29.435744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.435712    5533 generic.go:356] "Generic (PLEG): container finished" podID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerID="f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96" exitCode=0
Nov 03 21:22:29.435904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.435762    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"
Nov 03 21:22:29.435904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.435800    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" event={"ID":"9519b47f-00b3-4316-8620-0cf0b64af21d","Type":"ContainerDied","Data":"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"}
Nov 03 21:22:29.435904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.435863    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg" event={"ID":"9519b47f-00b3-4316-8620-0cf0b64af21d","Type":"ContainerDied","Data":"2cb3b0093b212fb4cd7834ede71842e05bf6cf622017636126130af2a31a7438"}
Nov 03 21:22:29.435904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.435893    5533 scope.go:122] "RemoveContainer" containerID="f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"
Nov 03 21:22:29.446481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.446462    5533 scope.go:122] "RemoveContainer" containerID="f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"
Nov 03 21:22:29.446784 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:22:29.446764    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96\": container with ID starting with f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96 not found: ID does not exist" containerID="f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"
Nov 03 21:22:29.446845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.446793    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96"} err="failed to get container status \"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96\": rpc error: code = NotFound desc = could not find container \"f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96\": container with ID starting with f440428e7355c39ba8e58f53969a6585471f6c8eb2995b53fd828f9110886b96 not found: ID does not exist"
Nov 03 21:22:29.593621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.593595    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:22:29.599227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.599204    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nm6xg"]
Nov 03 21:22:29.623393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.622801    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:22:29.627693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.627669    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy"
Nov 03 21:22:29.629824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.629798    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy"
Nov 03 21:22:29.630163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.630145    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" containerName="istio-proxy"
Nov 03 21:22:29.635970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.635946    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.681565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.681544    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:22:29.685573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.685550    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a77060\""
Nov 03 21:22:29.685573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.685550    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a77060-dockercfg-886p8\""
Nov 03 21:22:29.763184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.763156    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2gk\" (UniqueName: \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.763301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.763225    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.864166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.864139    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2gk\" (UniqueName: \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.864276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.864188    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.864883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.864866    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.922347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.922328    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2gk\" (UniqueName: \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\") pod \"payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:29.953395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:29.953369    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:30.319374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.319294    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:22:30.323904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.323882    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.335552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.335526    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:22:30.339432 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:22:30.339407    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398a8e11_5956_4904_9118_203967ac073c.slice/crio-dc5ac6a4ff68fcb2fb1ee39a9ff1a33acd8e726abc0acf93c91142df5bf68198 WatchSource:0}: Error finding container dc5ac6a4ff68fcb2fb1ee39a9ff1a33acd8e726abc0acf93c91142df5bf68198: Status 404 returned error can't find the container with id dc5ac6a4ff68fcb2fb1ee39a9ff1a33acd8e726abc0acf93c91142df5bf68198
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371350    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371429    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371540    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371592    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371624    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfqn\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371678    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371709    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.372046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.371747    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.425658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.415846    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:22:30.460671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.449850    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3d317d-openshift-default-dockercfg-lqz6r\""
Nov 03 21:22:30.461133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.461021    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" event={"ID":"398a8e11-5956-4904-9118-203967ac073c","Type":"ContainerStarted","Data":"dc5ac6a4ff68fcb2fb1ee39a9ff1a33acd8e726abc0acf93c91142df5bf68198"}
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.481985    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482057    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482119    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482163    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482211    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482238    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfqn\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482273    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482299    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.482336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.483811    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.484730    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.485311    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.485545    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.497878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.485787    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.513742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.512059    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.513742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.513678    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.514000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.513971    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9519b47f-00b3-4316-8620-0cf0b64af21d" path="/var/lib/kubelet/pods/9519b47f-00b3-4316-8620-0cf0b64af21d/volumes"
Nov 03 21:22:30.542693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.542604    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfqn\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.560353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.560321    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\") pod \"e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") " pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:30.639429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:30.639392    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:31.074374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.074345    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:22:31.076165 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:22:31.076130    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef42301c_9ba0_4ffd_83ce_14d30787e60d.slice/crio-82e2fd19a8919de54bf498ab0eb8bda61e3af8345a18ff70b98db2d8d70f6836 WatchSource:0}: Error finding container 82e2fd19a8919de54bf498ab0eb8bda61e3af8345a18ff70b98db2d8d70f6836: Status 404 returned error can't find the container with id 82e2fd19a8919de54bf498ab0eb8bda61e3af8345a18ff70b98db2d8d70f6836
Nov 03 21:22:31.079193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.079151    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:31.079301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.079226    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:31.079301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.079257    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:22:31.473690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.473415    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerStarted","Data":"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"}
Nov 03 21:22:31.473690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.473477    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerStarted","Data":"82e2fd19a8919de54bf498ab0eb8bda61e3af8345a18ff70b98db2d8d70f6836"}
Nov 03 21:22:31.475483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.475454    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" event={"ID":"398a8e11-5956-4904-9118-203967ac073c","Type":"ContainerStarted","Data":"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"}
Nov 03 21:22:31.475644 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.475514    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:31.587215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.587158    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podStartSLOduration=1.587145829 podStartE2EDuration="1.587145829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:31.586393939 +0000 UTC m=+2053.217707416" watchObservedRunningTime="2026-11-03 21:22:31.587145829 +0000 UTC m=+2053.218459306"
Nov 03 21:22:31.640679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.640653    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:31.699112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:31.699055    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" podStartSLOduration=2.6990436 podStartE2EDuration="2.6990436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:31.69582734 +0000 UTC m=+2053.327140816" watchObservedRunningTime="2026-11-03 21:22:31.6990436 +0000 UTC m=+2053.330357109"
Nov 03 21:22:32.645966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:32.645937    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:33.489259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:33.489219    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:33.490266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:33.490244    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:22:36.063009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:36.062975    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:22:36.063443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:36.063229    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy" containerID="cri-o://3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e" gracePeriod=30
Nov 03 21:22:38.966408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:38.966372    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:22:38.966860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:38.966652    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" containerID="cri-o://b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71" gracePeriod=30
Nov 03 21:22:40.207343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:40.207309    5533 patch_prober.go:29] interesting pod/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:22:40.207826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:40.207373    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:22:41.318816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.318791    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:22:41.450019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.449989    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450065    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450137    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450186    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450283    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450314    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:41.450406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450337    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450419    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:41.450515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450436    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450472    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450511    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\") pod \"995b0d00-5281-42a9-b8cd-9c544baab2cd\" (UID: \"995b0d00-5281-42a9-b8cd-9c544baab2cd\") "
Nov 03 21:22:41.450688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450563    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:41.450781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450702    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:41.450945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.450921    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:41.451096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.451081    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.451160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.451102    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995b0d00-5281-42a9-b8cd-9c544baab2cd-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.451160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.451116    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.451160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.451130    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.451160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.451143    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.452767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.452742    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:41.453020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.452993    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:41.453118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.453096    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:41.453163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.453124    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6" pod "995b0d00-5281-42a9-b8cd-9c544baab2cd" (UID: "995b0d00-5281-42a9-b8cd-9c544baab2cd"). InnerVolumeSpecName "kube-api-access-9shk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:41.542222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.542190    5533 generic.go:356] "Generic (PLEG): container finished" podID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerID="3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e" exitCode=0
Nov 03 21:22:41.542376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.542257    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"
Nov 03 21:22:41.542376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.542265    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" event={"ID":"995b0d00-5281-42a9-b8cd-9c544baab2cd","Type":"ContainerDied","Data":"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"}
Nov 03 21:22:41.542376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.542324    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs" event={"ID":"995b0d00-5281-42a9-b8cd-9c544baab2cd","Type":"ContainerDied","Data":"01fc674b5779e9299f2674668d9484a273fb654a52481894b9b2011024274882"}
Nov 03 21:22:41.542376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.542351    5533 scope.go:122] "RemoveContainer" containerID="3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"
Nov 03 21:22:41.552175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.552146    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shk6\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-kube-api-access-9shk6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.552295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.552178    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.552295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.552192    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.552295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.552204    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995b0d00-5281-42a9-b8cd-9c544baab2cd-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:41.553735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.553691    5533 scope.go:122] "RemoveContainer" containerID="3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"
Nov 03 21:22:41.554084 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:22:41.554053    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e\": container with ID starting with 3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e not found: ID does not exist" containerID="3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"
Nov 03 21:22:41.554197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.554091    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e"} err="failed to get container status \"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e\": rpc error: code = NotFound desc = could not find container \"3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e\": container with ID starting with 3c28c958f9456cf97236b6b08b38caf7c7b081fa6372d3fb226bd99be7fd243e not found: ID does not exist"
Nov 03 21:22:41.703505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.703430    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:22:41.706369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:41.706344    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-vxszs"]
Nov 03 21:22:42.531672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:42.531611    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" path="/var/lib/kubelet/pods/995b0d00-5281-42a9-b8cd-9c544baab2cd/volumes"
Nov 03 21:22:42.535864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:42.535836    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:22:42.798139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:42.798062    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:22:44.235594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.235542    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:22:44.388613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388585    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.388785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388711    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.388785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388744    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.388909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388794    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.388909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388826    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.388909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388857    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.389071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388925    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.389071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.388986    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:44.389071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389029    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.389235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389101    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\") pod \"2f7118dc-fcf0-4236-a976-3dac9fa643da\" (UID: \"2f7118dc-fcf0-4236-a976-3dac9fa643da\") "
Nov 03 21:22:44.389235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389155    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:44.389235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389172    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:44.389390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389269    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:44.389506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389402    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:44.389585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389535    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.389585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389555    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.389585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389567    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.389585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389576    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.389585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.389586    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f7118dc-fcf0-4236-a976-3dac9fa643da-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.391140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.391114    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:44.391481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.391456    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "kube-api-access-fd2tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:44.391602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.391480    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:44.391742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.391721    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo" pod "2f7118dc-fcf0-4236-a976-3dac9fa643da" (UID: "2f7118dc-fcf0-4236-a976-3dac9fa643da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:44.490402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.490183    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.490402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.490216    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2tc\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-kube-api-access-fd2tc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.490402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.490234    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.490402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.490255    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f7118dc-fcf0-4236-a976-3dac9fa643da-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:44.570392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.570362    5533 generic.go:356] "Generic (PLEG): container finished" podID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerID="b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71" exitCode=0
Nov 03 21:22:44.570528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.570413    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" event={"ID":"2f7118dc-fcf0-4236-a976-3dac9fa643da","Type":"ContainerDied","Data":"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"}
Nov 03 21:22:44.570528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.570483    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5" event={"ID":"2f7118dc-fcf0-4236-a976-3dac9fa643da","Type":"ContainerDied","Data":"324c4f97286f7e4a54af54b0567550bf45e09780500fc68282888132f4228151"}
Nov 03 21:22:44.570528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.570428    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"
Nov 03 21:22:44.570528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.570514    5533 scope.go:122] "RemoveContainer" containerID="b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"
Nov 03 21:22:44.580237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.580219    5533 scope.go:122] "RemoveContainer" containerID="b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"
Nov 03 21:22:44.580495 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:22:44.580475    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71\": container with ID starting with b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71 not found: ID does not exist" containerID="b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"
Nov 03 21:22:44.580558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.580505    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71"} err="failed to get container status \"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71\": rpc error: code = NotFound desc = could not find container \"b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71\": container with ID starting with b7fa70f19f549431a91e5eaac0e43c991f11dd986d4381bb2f6c1d76b6230b71 not found: ID does not exist"
Nov 03 21:22:44.728494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.728464    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:22:44.733095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:44.733066    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-thjx5"]
Nov 03 21:22:46.494474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:46.494439    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" path="/var/lib/kubelet/pods/2f7118dc-fcf0-4236-a976-3dac9fa643da/volumes"
Nov 03 21:22:49.262539 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:22:49.262492    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:22:56.429292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.429257    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:22:56.429928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.429509    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" podUID="aea0d2ed-44ad-4b68-af03-77e134f670ec" containerName="payload-pre-processing" containerID="cri-o://c4b1d143843f1dc3702ca78d10d916c231237fdb0bba1ac1e849a5a0f7461730" gracePeriod=30
Nov 03 21:22:56.664655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.661883    5533 generic.go:356] "Generic (PLEG): container finished" podID="aea0d2ed-44ad-4b68-af03-77e134f670ec" containerID="c4b1d143843f1dc3702ca78d10d916c231237fdb0bba1ac1e849a5a0f7461730" exitCode=0
Nov 03 21:22:56.664655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.661906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" event={"ID":"aea0d2ed-44ad-4b68-af03-77e134f670ec","Type":"ContainerDied","Data":"c4b1d143843f1dc3702ca78d10d916c231237fdb0bba1ac1e849a5a0f7461730"}
Nov 03 21:22:56.717901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.717874    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:22:56.736230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.736207    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\") pod \"aea0d2ed-44ad-4b68-af03-77e134f670ec\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") "
Nov 03 21:22:56.736323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.736282    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\" (UniqueName: \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\") pod \"aea0d2ed-44ad-4b68-af03-77e134f670ec\" (UID: \"aea0d2ed-44ad-4b68-af03-77e134f670ec\") "
Nov 03 21:22:56.736613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.736589    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume" pod "aea0d2ed-44ad-4b68-af03-77e134f670ec" (UID: "aea0d2ed-44ad-4b68-af03-77e134f670ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:56.738307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.738288    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr" pod "aea0d2ed-44ad-4b68-af03-77e134f670ec" (UID: "aea0d2ed-44ad-4b68-af03-77e134f670ec"). InnerVolumeSpecName "kube-api-access-m8mcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:56.837731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.837705    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea0d2ed-44ad-4b68-af03-77e134f670ec-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:56.837833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:56.837735    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mcr\" (UniqueName: \"kubernetes.io/projected/aea0d2ed-44ad-4b68-af03-77e134f670ec-kube-api-access-m8mcr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:57.670951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:57.670917    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"
Nov 03 21:22:57.671406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:57.670951    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp" event={"ID":"aea0d2ed-44ad-4b68-af03-77e134f670ec","Type":"ContainerDied","Data":"4bfab5b7f6a70206e4576e50801c03f01702e6e07de0bde5222c012736f8f453"}
Nov 03 21:22:57.671406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:57.671019    5533 scope.go:122] "RemoveContainer" containerID="c4b1d143843f1dc3702ca78d10d916c231237fdb0bba1ac1e849a5a0f7461730"
Nov 03 21:22:57.826361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:57.826329    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:22:57.833093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:57.833067    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-66513c-77b4bdb4-t5cdp"]
Nov 03 21:22:58.494741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:22:58.494713    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea0d2ed-44ad-4b68-af03-77e134f670ec" path="/var/lib/kubelet/pods/aea0d2ed-44ad-4b68-af03-77e134f670ec/volumes"
Nov 03 21:23:04.075752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:04.075670    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:23:04.076177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:04.075869    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy" containerID="cri-o://2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61" gracePeriod=30
Nov 03 21:23:09.343047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.342988    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:23:09.374040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374016    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374067    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374097    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374129    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374183    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374218    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374310    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374383    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374423    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\") pod \"e00acc07-571e-43cc-820d-1768a228c626\" (UID: \"e00acc07-571e-43cc-820d-1768a228c626\") "
Nov 03 21:23:09.374511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374482    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:09.374663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374571    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:09.374759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374735    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:09.374841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374818    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:09.374889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374850    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:09.375022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.374997    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.375166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.375025    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.375166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.375042    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00acc07-571e-43cc-820d-1768a228c626-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.375166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.375059    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.375166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.375081    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.376364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.376338    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:09.376798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.376775    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "kube-api-access-d84hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:09.376883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.376792    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:09.377089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.377065    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo" pod "e00acc07-571e-43cc-820d-1768a228c626" (UID: "e00acc07-571e-43cc-820d-1768a228c626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:23:09.476432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.476410    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.476432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.476434    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00acc07-571e-43cc-820d-1768a228c626-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.476570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.476444    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84hv\" (UniqueName: \"kubernetes.io/projected/e00acc07-571e-43cc-820d-1768a228c626-kube-api-access-d84hv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.476570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.476454    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00acc07-571e-43cc-820d-1768a228c626-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:09.750950 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.750918    5533 generic.go:356] "Generic (PLEG): container finished" podID="e00acc07-571e-43cc-820d-1768a228c626" containerID="2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61" exitCode=0
Nov 03 21:23:09.751170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.750952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerDied","Data":"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"}
Nov 03 21:23:09.751170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.750995    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"
Nov 03 21:23:09.751170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.751034    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd" event={"ID":"e00acc07-571e-43cc-820d-1768a228c626","Type":"ContainerDied","Data":"f087e9a463386318a09ae9b478e1a26779f359ec2288dc415adeaa20d4dd846f"}
Nov 03 21:23:09.751170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.751046    5533 scope.go:122] "RemoveContainer" containerID="2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"
Nov 03 21:23:09.762367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.762347    5533 scope.go:122] "RemoveContainer" containerID="ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"
Nov 03 21:23:09.773187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.773171    5533 scope.go:122] "RemoveContainer" containerID="2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"
Nov 03 21:23:09.773438 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:09.773418    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61\": container with ID starting with 2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61 not found: ID does not exist" containerID="2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"
Nov 03 21:23:09.773483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.773445    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61"} err="failed to get container status \"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61\": rpc error: code = NotFound desc = could not find container \"2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61\": container with ID starting with 2b86e61b25c8f5b062b643a4c212b3ecfb8562f27344181632ef921a1b2d0b61 not found: ID does not exist"
Nov 03 21:23:09.773483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.773463    5533 scope.go:122] "RemoveContainer" containerID="ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"
Nov 03 21:23:09.773725 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:09.773707    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672\": container with ID starting with ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672 not found: ID does not exist" containerID="ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"
Nov 03 21:23:09.773765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.773731    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672"} err="failed to get container status \"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672\": rpc error: code = NotFound desc = could not find container \"ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672\": container with ID starting with ad95996f61ea53d7e204c424e7d60b21dcb3b819b1542666915d69eb50a38672 not found: ID does not exist"
Nov 03 21:23:09.907012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.906988    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:23:09.910546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:09.910521    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-66513c-openshift-default-5c9c8f74c5-z2qrd"]
Nov 03 21:23:10.493832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:10.493803    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00acc07-571e-43cc-820d-1768a228c626" path="/var/lib/kubelet/pods/e00acc07-571e-43cc-820d-1768a228c626/volumes"
Nov 03 21:23:15.787356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.786541    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787431    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787455    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787470    5533 cpu_manager.go:414] "containerMap: removing container" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787478    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787494    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787502    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787512    5533 cpu_manager.go:414] "containerMap: removing container" podUID="aea0d2ed-44ad-4b68-af03-77e134f670ec" containerName="payload-pre-processing"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787519    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea0d2ed-44ad-4b68-af03-77e134f670ec" containerName="payload-pre-processing"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787540    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787548    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787752    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787769    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="995b0d00-5281-42a9-b8cd-9c544baab2cd" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787779    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7118dc-fcf0-4236-a976-3dac9fa643da" containerName="istio-proxy"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787789    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea0d2ed-44ad-4b68-af03-77e134f670ec" containerName="payload-pre-processing"
Nov 03 21:23:15.788341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.787800    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00acc07-571e-43cc-820d-1768a228c626" containerName="istio-proxy"
Nov 03 21:23:15.808696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.803798    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.863704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.863663    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:23:15.904729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.904701    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1dda2c-openshift-default-dockercfg-gdslr\""
Nov 03 21:23:15.946074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946039    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946106    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946240    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946356    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946436    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946510    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946591    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946619    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:15.946702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:15.946672    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5qr\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048213    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048284    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048303    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048325    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048366    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048390    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048506    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5qr\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048562    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048668    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048787    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.048995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048838    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.049052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.048993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.049052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.049023    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.049209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.049181    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.050780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.050755    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.051040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.051023    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.108521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.108495    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.109015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.108999    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5qr\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\") pod \"e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") " pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.123888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.123866    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:16.544864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.544830    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:23:16.545871 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:23:16.545836    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2662e667_b1d9_4b9d_824f_0d8cec0305bc.slice/crio-07b66a0f78635bd4b2ecd8fa349cdfd87974c80135af6d31b717eeafdc8f2035 WatchSource:0}: Error finding container 07b66a0f78635bd4b2ecd8fa349cdfd87974c80135af6d31b717eeafdc8f2035: Status 404 returned error can't find the container with id 07b66a0f78635bd4b2ecd8fa349cdfd87974c80135af6d31b717eeafdc8f2035
Nov 03 21:23:16.548533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.548491    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:16.548618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.548579    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:16.548685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.548618    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:16.819471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.819385    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerStarted","Data":"3985ff90840d5c858cdc8f53dac3c028660a85aeebdf31dd420f79c8d6adc980"}
Nov 03 21:23:16.819471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.819444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerStarted","Data":"07b66a0f78635bd4b2ecd8fa349cdfd87974c80135af6d31b717eeafdc8f2035"}
Nov 03 21:23:16.935789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:16.935740    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podStartSLOduration=1.93572467 podStartE2EDuration="1.93572467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:16.93278177 +0000 UTC m=+2098.564095248" watchObservedRunningTime="2026-11-03 21:23:16.93572467 +0000 UTC m=+2098.567038223"
Nov 03 21:23:17.124761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:17.124671    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:17.130559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:17.130529    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:17.826267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:17.826236    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:17.827080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:17.827055    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:23:18.893369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:18.893326    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:23:18.906271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:18.906243    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:23:18.910604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:18.910577    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:23:18.920758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:18.920735    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:23:19.260751 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:19.260714    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:23:19.655769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:19.655689    5533 scope.go:122] "RemoveContainer" containerID="c79da4ba0eb710d4c9e6ab495b9754423a53fafbb9577f252874efca5b360462"
Nov 03 21:23:23.964496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:23.964463    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:23:23.970291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:23.970270    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.025568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.025527    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:23:24.053512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.053481    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.053728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.053548    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cgz\" (UniqueName: \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.084978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.084939    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1dda2c-dockercfg-8qn7t\""
Nov 03 21:23:24.116394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.104999    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1dda2c\""
Nov 03 21:23:24.140077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.140031    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:23:24.163686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.157425    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.163686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.157534    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cgz\" (UniqueName: \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.165691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.165623    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.171581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.171547    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.186763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.181317    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:23:24.215261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.215209    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cgz\" (UniqueName: \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\") pod \"payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.259158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.259128    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtht\" (UniqueName: \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.259297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.259253    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.286945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.286920    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.360235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.360099    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtht\" (UniqueName: \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.360235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.360227    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.361360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.361182    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.418575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.418550    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtht\" (UniqueName: \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\") pod \"payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.495922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.495868    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:24.659247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.659217    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:23:24.661678 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:23:24.661611    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0e87f7_e4cf_4f57_a9f3_7e870c3d0bdc.slice/crio-5b27a4776aee1fe02c1312e4d3757a3bfb22b1b974feb67a1c5257a15d537297 WatchSource:0}: Error finding container 5b27a4776aee1fe02c1312e4d3757a3bfb22b1b974feb67a1c5257a15d537297: Status 404 returned error can't find the container with id 5b27a4776aee1fe02c1312e4d3757a3bfb22b1b974feb67a1c5257a15d537297
Nov 03 21:23:24.869516 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:23:24.869486    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a372f5_8b6d_4dba_ae9c_63c6b528e82e.slice/crio-19a261c94d88f4c72ac7e7ed5ee001b2cd2aec2f6c8e32c9e98d85856e4f2633 WatchSource:0}: Error finding container 19a261c94d88f4c72ac7e7ed5ee001b2cd2aec2f6c8e32c9e98d85856e4f2633: Status 404 returned error can't find the container with id 19a261c94d88f4c72ac7e7ed5ee001b2cd2aec2f6c8e32c9e98d85856e4f2633
Nov 03 21:23:24.870179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.870050    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:23:24.908379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.908350    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" event={"ID":"57a372f5-8b6d-4dba-ae9c-63c6b528e82e","Type":"ContainerStarted","Data":"19a261c94d88f4c72ac7e7ed5ee001b2cd2aec2f6c8e32c9e98d85856e4f2633"}
Nov 03 21:23:24.909999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.909970    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" event={"ID":"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc","Type":"ContainerStarted","Data":"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"}
Nov 03 21:23:24.910093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.910029    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:24.910093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:24.910054    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" event={"ID":"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc","Type":"ContainerStarted","Data":"5b27a4776aee1fe02c1312e4d3757a3bfb22b1b974feb67a1c5257a15d537297"}
Nov 03 21:23:25.022457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:25.022412    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" podStartSLOduration=2.022399071 podStartE2EDuration="2.022399071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:25.019464269 +0000 UTC m=+2106.650777760" watchObservedRunningTime="2026-11-03 21:23:25.022399071 +0000 UTC m=+2106.653712547"
Nov 03 21:23:25.917700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:25.917664    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" event={"ID":"57a372f5-8b6d-4dba-ae9c-63c6b528e82e","Type":"ContainerStarted","Data":"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"}
Nov 03 21:23:25.917905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:25.917764    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:26.028344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:26.028285    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" podStartSLOduration=2.028271141 podStartE2EDuration="2.028271141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:26.026252488 +0000 UTC m=+2107.657565964" watchObservedRunningTime="2026-11-03 21:23:26.028271141 +0000 UTC m=+2107.659584618"
Nov 03 21:23:29.537979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:29.537948    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:23:34.805438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:34.805388    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:23:34.805977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:34.805688    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" podUID="296697e9-2fc1-49a0-a004-5176cf8d125a" containerName="payload-processing" containerID="cri-o://0d92f2266e4089b2f05f7b1bdf219d1c30af759e7895e2682c1be63c1bc21a6d" gracePeriod=30
Nov 03 21:23:35.003188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:34.998737    5533 generic.go:356] "Generic (PLEG): container finished" podID="296697e9-2fc1-49a0-a004-5176cf8d125a" containerID="0d92f2266e4089b2f05f7b1bdf219d1c30af759e7895e2682c1be63c1bc21a6d" exitCode=0
Nov 03 21:23:35.003188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:34.998848    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" event={"ID":"296697e9-2fc1-49a0-a004-5176cf8d125a","Type":"ContainerDied","Data":"0d92f2266e4089b2f05f7b1bdf219d1c30af759e7895e2682c1be63c1bc21a6d"}
Nov 03 21:23:35.118710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.118683    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:23:35.172922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.172893    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\" (UniqueName: \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\") pod \"296697e9-2fc1-49a0-a004-5176cf8d125a\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") "
Nov 03 21:23:35.173097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.172930    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\") pod \"296697e9-2fc1-49a0-a004-5176cf8d125a\" (UID: \"296697e9-2fc1-49a0-a004-5176cf8d125a\") "
Nov 03 21:23:35.173447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.173408    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume" pod "296697e9-2fc1-49a0-a004-5176cf8d125a" (UID: "296697e9-2fc1-49a0-a004-5176cf8d125a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:35.175154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.175126    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v" pod "296697e9-2fc1-49a0-a004-5176cf8d125a" (UID: "296697e9-2fc1-49a0-a004-5176cf8d125a"). InnerVolumeSpecName "kube-api-access-hzn9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:35.274300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.274272    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzn9v\" (UniqueName: \"kubernetes.io/projected/296697e9-2fc1-49a0-a004-5176cf8d125a-kube-api-access-hzn9v\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:35.274300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.274300    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296697e9-2fc1-49a0-a004-5176cf8d125a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:35.920844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:35.920816    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:23:36.007161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.007131    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"
Nov 03 21:23:36.007315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.007131    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv" event={"ID":"296697e9-2fc1-49a0-a004-5176cf8d125a","Type":"ContainerDied","Data":"082b3256b127b26cfb384e05e9cf90db6664efe70600822e915c41fc8da7e73e"}
Nov 03 21:23:36.007315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.007271    5533 scope.go:122] "RemoveContainer" containerID="0d92f2266e4089b2f05f7b1bdf219d1c30af759e7895e2682c1be63c1bc21a6d"
Nov 03 21:23:36.136497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.136467    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:23:36.142776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.142751    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-812ef9-5f5f876db6-2jqpv"]
Nov 03 21:23:36.494847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.494810    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296697e9-2fc1-49a0-a004-5176cf8d125a" path="/var/lib/kubelet/pods/296697e9-2fc1-49a0-a004-5176cf8d125a/volumes"
Nov 03 21:23:36.926648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:36.926616    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:23:46.296796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:46.296765    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:23:46.297205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:46.296965    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy" containerID="cri-o://22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278" gracePeriod=30
Nov 03 21:23:49.262548 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:49.262507    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:23:51.582159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.582136    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:23:51.614922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.614886    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.614933    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615032    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615077    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615120    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615157    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615199    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615291    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:51.615351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615316    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615357    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\") pod \"44557d39-db06-4b1c-b42a-8168134023b5\" (UID: \"44557d39-db06-4b1c-b42a-8168134023b5\") "
Nov 03 21:23:51.615516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615463    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:51.615817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615758    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:23:51.615914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615858    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.615914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615875    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.615914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.615885    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44557d39-db06-4b1c-b42a-8168134023b5-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.616284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.616250    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:51.616417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.616360    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:51.617501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.617475    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:51.618047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.617998    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:23:51.618386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.618369    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "kube-api-access-8fhdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:51.618680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.618655    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo" pod "44557d39-db06-4b1c-b42a-8168134023b5" (UID: "44557d39-db06-4b1c-b42a-8168134023b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:23:51.717401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717367    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.717401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717398    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhdr\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-kube-api-access-8fhdr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.717547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717413    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44557d39-db06-4b1c-b42a-8168134023b5-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.717547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717425    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44557d39-db06-4b1c-b42a-8168134023b5-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.717547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717439    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:51.717547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:51.717451    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44557d39-db06-4b1c-b42a-8168134023b5-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:23:52.129703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.129656    5533 generic.go:356] "Generic (PLEG): container finished" podID="44557d39-db06-4b1c-b42a-8168134023b5" containerID="22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278" exitCode=0
Nov 03 21:23:52.129703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.129687    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerDied","Data":"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"}
Nov 03 21:23:52.129920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.129713    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"
Nov 03 21:23:52.129920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.129747    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb" event={"ID":"44557d39-db06-4b1c-b42a-8168134023b5","Type":"ContainerDied","Data":"c1e0f349dbf318ca878f563aaaea6df746ee518d79ce4ae6328e35bdd2d2026c"}
Nov 03 21:23:52.129920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.129771    5533 scope.go:122] "RemoveContainer" containerID="22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"
Nov 03 21:23:52.140729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.140712    5533 scope.go:122] "RemoveContainer" containerID="cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"
Nov 03 21:23:52.149476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.149458    5533 scope.go:122] "RemoveContainer" containerID="22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"
Nov 03 21:23:52.149759 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:52.149736    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278\": container with ID starting with 22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278 not found: ID does not exist" containerID="22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"
Nov 03 21:23:52.149828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.149765    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278"} err="failed to get container status \"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278\": rpc error: code = NotFound desc = could not find container \"22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278\": container with ID starting with 22b77b7ee00fc433cdacc95801f0e645b49ac32da32717a431dc8ab464503278 not found: ID does not exist"
Nov 03 21:23:52.149828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.149781    5533 scope.go:122] "RemoveContainer" containerID="cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"
Nov 03 21:23:52.150031 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:23:52.150013    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330\": container with ID starting with cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330 not found: ID does not exist" containerID="cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"
Nov 03 21:23:52.150078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.150039    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330"} err="failed to get container status \"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330\": rpc error: code = NotFound desc = could not find container \"cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330\": container with ID starting with cb2bae367bbea8ca10e975f39eeb1431f95bdc18d002fb73d233e791ac62e330 not found: ID does not exist"
Nov 03 21:23:52.286267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.286246    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:23:52.332853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.332823    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-812ef9-openshift-default-6c8b8c9dbc-kgnvb"]
Nov 03 21:23:52.501338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:52.501303    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44557d39-db06-4b1c-b42a-8168134023b5" path="/var/lib/kubelet/pods/44557d39-db06-4b1c-b42a-8168134023b5/volumes"
Nov 03 21:23:56.797208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.797172    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:23:56.798063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798039    5533 cpu_manager.go:414] "containerMap: removing container" podUID="296697e9-2fc1-49a0-a004-5176cf8d125a" containerName="payload-processing"
Nov 03 21:23:56.798170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798064    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296697e9-2fc1-49a0-a004-5176cf8d125a" containerName="payload-processing"
Nov 03 21:23:56.798170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798129    5533 cpu_manager.go:414] "containerMap: removing container" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.798170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798137    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.798170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798167    5533 cpu_manager.go:414] "containerMap: removing container" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.798381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798176    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.798381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798307    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.798381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.798323    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="296697e9-2fc1-49a0-a004-5176cf8d125a" containerName="payload-processing"
Nov 03 21:23:56.803934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.803914    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.854886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.854861    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:23:56.877885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.877857    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.878023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.877999    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.897953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.897901    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b5c926-dockercfg-z5psl\""
Nov 03 21:23:56.899790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.899754    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b5c926\""
Nov 03 21:23:56.982494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.979682    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:23:56.982494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.980617    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="44557d39-db06-4b1c-b42a-8168134023b5" containerName="istio-proxy"
Nov 03 21:23:56.986291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.984506    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.986291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.985822    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.986291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.985540    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:56.992443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:56.990102    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.016211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.016165    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:23:57.044264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.044233    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\") pod \"payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:57.086605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.086508    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.086605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.086583    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vxd\" (UniqueName: \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.116760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.116735    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:57.191659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.189388    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.191659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.189444    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vxd\" (UniqueName: \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.191659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.190628    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.247445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.247415    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vxd\" (UniqueName: \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\") pod \"payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.310799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.310772    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:57.500917 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:23:57.500889    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a49d4c_5570_42ae_b76e_329d4e226d9a.slice/crio-f89aec861378bd0367b1492fc21ea895fe22ea286a08ab5bd4a45fde080aac77 WatchSource:0}: Error finding container f89aec861378bd0367b1492fc21ea895fe22ea286a08ab5bd4a45fde080aac77: Status 404 returned error can't find the container with id f89aec861378bd0367b1492fc21ea895fe22ea286a08ab5bd4a45fde080aac77
Nov 03 21:23:57.501035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.500922    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:23:57.684845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:57.684815    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:23:57.686149 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:23:57.686050    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fda55b2_0134_4b76_87ca_522cacca8c84.slice/crio-27311286f75cce68b6dd68b7203fc8e7561836c74f1cec93a7edc805eae1072b WatchSource:0}: Error finding container 27311286f75cce68b6dd68b7203fc8e7561836c74f1cec93a7edc805eae1072b: Status 404 returned error can't find the container with id 27311286f75cce68b6dd68b7203fc8e7561836c74f1cec93a7edc805eae1072b
Nov 03 21:23:58.030700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.030585    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:23:58.031120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.030856    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy" containerID="cri-o://72620d9489c5ec37b22c34c01738cb8a124c6e94930d0a6a1656a3a04869a0e2" gracePeriod=30
Nov 03 21:23:58.208584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.206776    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" event={"ID":"1fda55b2-0134-4b76-87ca-522cacca8c84","Type":"ContainerStarted","Data":"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"}
Nov 03 21:23:58.208584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.206847    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" event={"ID":"1fda55b2-0134-4b76-87ca-522cacca8c84","Type":"ContainerStarted","Data":"27311286f75cce68b6dd68b7203fc8e7561836c74f1cec93a7edc805eae1072b"}
Nov 03 21:23:58.208584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.207821    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:23:58.211300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.211273    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" event={"ID":"66a49d4c-5570-42ae-b76e-329d4e226d9a","Type":"ContainerStarted","Data":"90537bbef84f5968e4160aa0c40f1f9a25c6d52cccc17db04c6f703eee478ef9"}
Nov 03 21:23:58.211555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.211515    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" event={"ID":"66a49d4c-5570-42ae-b76e-329d4e226d9a","Type":"ContainerStarted","Data":"f89aec861378bd0367b1492fc21ea895fe22ea286a08ab5bd4a45fde080aac77"}
Nov 03 21:23:58.211919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.211900    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:23:58.320106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.320012    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" podStartSLOduration=2.319996451 podStartE2EDuration="2.319996451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:58.316840665 +0000 UTC m=+2139.948154142" watchObservedRunningTime="2026-11-03 21:23:58.319996451 +0000 UTC m=+2139.951309930"
Nov 03 21:23:58.427220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:23:58.427165    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" podStartSLOduration=2.427150646 podStartE2EDuration="2.427150646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:58.425798047 +0000 UTC m=+2140.057111524" watchObservedRunningTime="2026-11-03 21:23:58.427150646 +0000 UTC m=+2140.058464123"
Nov 03 21:24:03.294984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.294945    5533 generic.go:356] "Generic (PLEG): container finished" podID="c319af1b-55d4-4175-8425-fdb04cae767f" containerID="72620d9489c5ec37b22c34c01738cb8a124c6e94930d0a6a1656a3a04869a0e2" exitCode=0
Nov 03 21:24:03.295510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.294997    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerDied","Data":"72620d9489c5ec37b22c34c01738cb8a124c6e94930d0a6a1656a3a04869a0e2"}
Nov 03 21:24:03.295510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.295087    5533 scope.go:122] "RemoveContainer" containerID="79924a3324e8395e18564fef1c7ef1d6a6dd4dd825977c7135d334f5491bc9f3"
Nov 03 21:24:03.390902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.390872    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:24:03.486140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486107    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486157    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486210    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486302    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486341    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486369    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486415    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486445    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486481    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\") pod \"c319af1b-55d4-4175-8425-fdb04cae767f\" (UID: \"c319af1b-55d4-4175-8425-fdb04cae767f\") "
Nov 03 21:24:03.486726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486592    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:24:03.486726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486667    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:24:03.486899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486868    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:24:03.486994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.486971    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:03.487091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.487068    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:24:03.487153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.487079    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.487153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.487112    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.487153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.487129    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.488925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.488735    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:24:03.489039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.488991    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:24:03.489320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.489297    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:03.489320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.489306    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7" pod "c319af1b-55d4-4175-8425-fdb04cae767f" (UID: "c319af1b-55d4-4175-8425-fdb04cae767f"). InnerVolumeSpecName "kube-api-access-xvbw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:03.588064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.587977    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.588064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.588013    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319af1b-55d4-4175-8425-fdb04cae767f-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.588064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.588030    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.588064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.588044    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319af1b-55d4-4175-8425-fdb04cae767f-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.588064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.588060    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbw7\" (UniqueName: \"kubernetes.io/projected/c319af1b-55d4-4175-8425-fdb04cae767f-kube-api-access-xvbw7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:03.588458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:03.588074    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319af1b-55d4-4175-8425-fdb04cae767f-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:04.304499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:04.304450    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"
Nov 03 21:24:04.305011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:04.304493    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll" event={"ID":"c319af1b-55d4-4175-8425-fdb04cae767f","Type":"ContainerDied","Data":"a4304c8368ef058fb2ff2a77f6a08d71fe4aa6e97927940a986a11684f78af1e"}
Nov 03 21:24:04.305011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:04.304556    5533 scope.go:122] "RemoveContainer" containerID="72620d9489c5ec37b22c34c01738cb8a124c6e94930d0a6a1656a3a04869a0e2"
Nov 03 21:24:04.507534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:04.507492    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:24:04.571708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:04.567945    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3e1b49-openshift-default-69c6d5454f-vx8ll"]
Nov 03 21:24:05.384871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:05.384826    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:24:05.488419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:05.488384    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:24:06.497383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:06.497339    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" path="/var/lib/kubelet/pods/c319af1b-55d4-4175-8425-fdb04cae767f/volumes"
Nov 03 21:24:09.046584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:09.046550    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"]
Nov 03 21:24:09.092907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:09.092874    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4842a9ae-r79kc"]
Nov 03 21:24:09.225835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:09.225806    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:24:10.235342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.235314    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:24:10.347290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.347260    5533 generic.go:356] "Generic (PLEG): container finished" podID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerID="c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe" exitCode=0
Nov 03 21:24:10.347463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.347340    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerDied","Data":"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"}
Nov 03 21:24:10.347757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.347741    5533 scope.go:122] "RemoveContainer" containerID="c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"
Nov 03 21:24:10.349171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.349141    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:10.349284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.349217    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:10.349284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.349259    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:10.385491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.385446    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" podUID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" containerName="payload-processing" containerID="cri-o://642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9" gracePeriod=25
Nov 03 21:24:10.490853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.490763    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" podUID="cb8f1b41-9509-44c7-8011-5b178e219613" containerName="payload-pre-processing" containerID="cri-o://73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0" gracePeriod=25
Nov 03 21:24:10.499253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.499225    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf2a840-049c-46b8-af0f-e0a3a9ff2a04" path="/var/lib/kubelet/pods/0cf2a840-049c-46b8-af0f-e0a3a9ff2a04/volumes"
Nov 03 21:24:10.695389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.695214    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:24:10.807771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.807741    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\" (UniqueName: \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\") pod \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") "
Nov 03 21:24:10.808667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.807942    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\") pod \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\" (UID: \"1e604671-7acb-4eb6-bc95-2ff428cb3c13\") "
Nov 03 21:24:10.808809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.808704    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume" pod "1e604671-7acb-4eb6-bc95-2ff428cb3c13" (UID: "1e604671-7acb-4eb6-bc95-2ff428cb3c13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:10.813065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.811267    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq" pod "1e604671-7acb-4eb6-bc95-2ff428cb3c13" (UID: "1e604671-7acb-4eb6-bc95-2ff428cb3c13"). InnerVolumeSpecName "kube-api-access-l49tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:10.843186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.843162    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:24:10.909688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.909622    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\") pod \"cb8f1b41-9509-44c7-8011-5b178e219613\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") "
Nov 03 21:24:10.909870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.909714    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\" (UniqueName: \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\") pod \"cb8f1b41-9509-44c7-8011-5b178e219613\" (UID: \"cb8f1b41-9509-44c7-8011-5b178e219613\") "
Nov 03 21:24:10.910025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.910011    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49tq\" (UniqueName: \"kubernetes.io/projected/1e604671-7acb-4eb6-bc95-2ff428cb3c13-kube-api-access-l49tq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:10.910083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.910026    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e604671-7acb-4eb6-bc95-2ff428cb3c13-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:10.910083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.910024    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume" pod "cb8f1b41-9509-44c7-8011-5b178e219613" (UID: "cb8f1b41-9509-44c7-8011-5b178e219613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:10.911736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:10.911707    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj" pod "cb8f1b41-9509-44c7-8011-5b178e219613" (UID: "cb8f1b41-9509-44c7-8011-5b178e219613"). InnerVolumeSpecName "kube-api-access-vvfnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:11.011055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.011023    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8f1b41-9509-44c7-8011-5b178e219613-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:11.011055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.011051    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfnj\" (UniqueName: \"kubernetes.io/projected/cb8f1b41-9509-44c7-8011-5b178e219613-kube-api-access-vvfnj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:11.389659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.386866    5533 generic.go:356] "Generic (PLEG): container finished" podID="cb8f1b41-9509-44c7-8011-5b178e219613" containerID="73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0" exitCode=0
Nov 03 21:24:11.389659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.387025    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" event={"ID":"cb8f1b41-9509-44c7-8011-5b178e219613","Type":"ContainerDied","Data":"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"}
Nov 03 21:24:11.389659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.387093    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx" event={"ID":"cb8f1b41-9509-44c7-8011-5b178e219613","Type":"ContainerDied","Data":"d10fae28f83a6c981889f093c18befe6fa5baa75cabd4da259e7eae567a98637"}
Nov 03 21:24:11.389659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.387134    5533 scope.go:122] "RemoveContainer" containerID="73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"
Nov 03 21:24:11.389659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.387285    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"
Nov 03 21:24:11.414034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.413997    5533 generic.go:356] "Generic (PLEG): container finished" podID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" containerID="642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9" exitCode=0
Nov 03 21:24:11.414234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.414211    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" event={"ID":"1e604671-7acb-4eb6-bc95-2ff428cb3c13","Type":"ContainerDied","Data":"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"}
Nov 03 21:24:11.414297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.414271    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5" event={"ID":"1e604671-7acb-4eb6-bc95-2ff428cb3c13","Type":"ContainerDied","Data":"46d0bbc8e6fd9d4ca8038d62589307d4af063eee7f426caad613b957bbadb512"}
Nov 03 21:24:11.414362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.414305    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"
Nov 03 21:24:11.417212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.417182    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerStarted","Data":"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"}
Nov 03 21:24:11.422545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.422524    5533 scope.go:122] "RemoveContainer" containerID="73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"
Nov 03 21:24:11.422879 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:24:11.422855    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0\": container with ID starting with 73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0 not found: ID does not exist" containerID="73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"
Nov 03 21:24:11.422954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.422891    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0"} err="failed to get container status \"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0\": rpc error: code = NotFound desc = could not find container \"73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0\": container with ID starting with 73dee74adfddf3ad438a1d6f1fd6cdddbe027cd4e60880ca35a6853f446714f0 not found: ID does not exist"
Nov 03 21:24:11.422954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.422916    5533 scope.go:122] "RemoveContainer" containerID="642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"
Nov 03 21:24:11.432058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.432036    5533 scope.go:122] "RemoveContainer" containerID="642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"
Nov 03 21:24:11.432331 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:24:11.432313    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9\": container with ID starting with 642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9 not found: ID does not exist" containerID="642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"
Nov 03 21:24:11.432394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.432338    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9"} err="failed to get container status \"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9\": rpc error: code = NotFound desc = could not find container \"642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9\": container with ID starting with 642587c1fe0643da34b6b21411a07eaac248bdcad26f00b4e9872e9f7c46d1d9 not found: ID does not exist"
Nov 03 21:24:11.578293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.578220    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:24:11.585008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.584977    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7af9dd-b9b979679-f8cdx"]
Nov 03 21:24:11.640605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.640529    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:24:11.640758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.640671    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 21:24:11.640758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.640714    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 21:24:11.679654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.679058    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:24:11.683280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:11.683254    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7af9dd-7bc9687548-lbwk5"]
Nov 03 21:24:12.498366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:12.498333    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" path="/var/lib/kubelet/pods/1e604671-7acb-4eb6-bc95-2ff428cb3c13/volumes"
Nov 03 21:24:12.498758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:12.498744    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8f1b41-9509-44c7-8011-5b178e219613" path="/var/lib/kubelet/pods/cb8f1b41-9509-44c7-8011-5b178e219613/volumes"
Nov 03 21:24:12.640119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:12.640083    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 21:24:12.640262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:12.640135    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 21:24:13.639950 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:13.639917    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 21:24:13.640330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:13.639972    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 21:24:14.639892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:14.639860    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 21:24:14.640073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:14.639909    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 21:24:15.639946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:15.639908    5533 patch_prober.go:29] interesting pod/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 21:24:15.640111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:15.639969    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 21:24:16.646122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:16.646089    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:24:16.646533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:16.646388    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:24:16.647016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:16.646996    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:24:19.260591 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:24:19.260555    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:24:19.715004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:19.714972    5533 scope.go:122] "RemoveContainer" containerID="4ff4d261e0c7fd2ccbb8c041a816fa46d9964a911451ed1575953a4ea83c5710"
Nov 03 21:24:31.118855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:31.118817    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:24:40.630940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.630863    5533 generic.go:356] "Generic (PLEG): container finished" podID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerID="3985ff90840d5c858cdc8f53dac3c028660a85aeebdf31dd420f79c8d6adc980" exitCode=0
Nov 03 21:24:40.630940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.630931    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerDied","Data":"3985ff90840d5c858cdc8f53dac3c028660a85aeebdf31dd420f79c8d6adc980"}
Nov 03 21:24:40.631366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.631294    5533 scope.go:122] "RemoveContainer" containerID="3985ff90840d5c858cdc8f53dac3c028660a85aeebdf31dd420f79c8d6adc980"
Nov 03 21:24:40.632707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.632681    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:40.632777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.632748    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:40.632817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:40.632780    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:24:41.639416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:41.639378    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerStarted","Data":"7ebd4383fbfd308a75752e1ee580b45fbc5794105acaff825ec526d01c7f947f"}
Nov 03 21:24:42.124225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:42.124194    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:24:42.124413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:42.124363    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 21:24:42.124467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:42.124416    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 21:24:43.124312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:43.124282    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 21:24:43.124709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:43.124331    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 21:24:44.125055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:44.125019    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 21:24:44.125427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:44.125073    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 21:24:45.124694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:45.124657    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 21:24:45.124869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:45.124733    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 21:24:46.124243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:46.124211    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 21:24:46.124648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:46.124273    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 21:24:47.128553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:47.128522    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:24:47.129018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:47.128998    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:24:47.129600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:24:47.129581    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:24:49.261787 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:24:49.261746    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:25:00.496691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.496625    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:25:00.497414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497392    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" containerName="payload-processing"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497415    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" containerName="payload-processing"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497439    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497448    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497461    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cb8f1b41-9509-44c7-8011-5b178e219613" containerName="payload-pre-processing"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497469    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8f1b41-9509-44c7-8011-5b178e219613" containerName="payload-pre-processing"
Nov 03 21:25:00.497498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497499    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497507    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497671    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497686    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8f1b41-9509-44c7-8011-5b178e219613" containerName="payload-pre-processing"
Nov 03 21:25:00.497854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497699    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c319af1b-55d4-4175-8425-fdb04cae767f" containerName="istio-proxy"
Nov 03 21:25:00.497854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.497717    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e604671-7acb-4eb6-bc95-2ff428cb3c13" containerName="payload-processing"
Nov 03 21:25:00.501687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.501667    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.546749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.546728    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:25:00.612578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.590961    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fe2df9\""
Nov 03 21:25:00.612578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.591133    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fe2df9-dockercfg-tkx2v\""
Nov 03 21:25:00.646207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.645223    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.646207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.645302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46krj\" (UniqueName: \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.660305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.655396    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:25:00.676062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.676037    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:00.698214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.698182    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:25:00.746688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.746621    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.746870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.746713    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46krj\" (UniqueName: \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.747408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.747388    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.804832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.804804    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46krj\" (UniqueName: \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\") pod \"payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.815644 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.815616    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:00.848200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.848170    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5pbx\" (UniqueName: \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:00.848324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.848283    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:00.949739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.949701    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5pbx\" (UniqueName: \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:00.949893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.949792    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:00.950507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:00.950484    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:01.007287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.007218    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5pbx\" (UniqueName: \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\") pod \"payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:01.196212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.196181    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:25:01.291486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.291459    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:01.671866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.671831    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:25:01.672330 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:25:01.671844    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda662eef1_9966_4d85_99cb_ffb19ea06881.slice/crio-83fe2c9063bb4297ca8276ac4ef99cd1c70a1151da714904571bb5f64ac9d518 WatchSource:0}: Error finding container 83fe2c9063bb4297ca8276ac4ef99cd1c70a1151da714904571bb5f64ac9d518: Status 404 returned error can't find the container with id 83fe2c9063bb4297ca8276ac4ef99cd1c70a1151da714904571bb5f64ac9d518
Nov 03 21:25:01.809363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.809331    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" event={"ID":"a0f81fc5-2722-4a35-9bfa-70ad571daee0","Type":"ContainerStarted","Data":"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"}
Nov 03 21:25:01.809505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.809389    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" event={"ID":"a0f81fc5-2722-4a35-9bfa-70ad571daee0","Type":"ContainerStarted","Data":"119b244000667dfa81b250c823e97325dbf0dba0242ced34e8004960cb817bb1"}
Nov 03 21:25:01.809569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.809540    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:01.811342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.811297    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" event={"ID":"a662eef1-9966-4d85-99cb-ffb19ea06881","Type":"ContainerStarted","Data":"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"}
Nov 03 21:25:01.811463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.811351    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:01.811463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:01.811375    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" event={"ID":"a662eef1-9966-4d85-99cb-ffb19ea06881","Type":"ContainerStarted","Data":"83fe2c9063bb4297ca8276ac4ef99cd1c70a1151da714904571bb5f64ac9d518"}
Nov 03 21:25:02.041798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:02.039244    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" podStartSLOduration=2.039223004 podStartE2EDuration="2.039223004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:02.037258767 +0000 UTC m=+2203.668572269" watchObservedRunningTime="2026-11-03 21:25:02.039223004 +0000 UTC m=+2203.670536482"
Nov 03 21:25:02.044506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:02.044431    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" podStartSLOduration=2.044411145 podStartE2EDuration="2.044411145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:01.92797296 +0000 UTC m=+2203.559286462" watchObservedRunningTime="2026-11-03 21:25:02.044411145 +0000 UTC m=+2203.675724624"
Nov 03 21:25:03.632812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:03.632777    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:25:12.829542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:12.829513    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:25:12.830114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:12.829712    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:25:19.263325 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:25:19.263289    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:25:19.766435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:19.766404    5533 scope.go:122] "RemoveContainer" containerID="858a21881c759443a295636c5f80bf8c80a2f46d1717a9be84af04ec00d44aaf"
Nov 03 21:25:28.124049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.124004    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"]
Nov 03 21:25:28.129575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.129555    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.180574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.180539    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.180775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.180735    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89f99\" (UniqueName: \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.180956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.180806    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.182618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.182597    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"]
Nov 03 21:25:28.225574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.225548    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:25:28.225721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.225607    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:25:28.281978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.281953    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.282105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.282088    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.282210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.282194    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89f99\" (UniqueName: \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.282719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.282702    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.284602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.284579    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.341008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.340979    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89f99\" (UniqueName: \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\") pod \"maas-api-revoke-sub-3597afd9-nw7nf\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.443205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.443170    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:28.822720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.822696    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"]
Nov 03 21:25:28.824089 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:25:28.824062    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1363be_79cd_4103_a588_5a87a285efb1.slice/crio-d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5 WatchSource:0}: Error finding container d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5: Status 404 returned error can't find the container with id d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5
Nov 03 21:25:28.997526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:28.997493    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf" event={"ID":"7c1363be-79cd-4103-a588-5a87a285efb1","Type":"ContainerStarted","Data":"d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5"}
Nov 03 21:25:30.005794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:30.005698    5533 generic.go:356] "Generic (PLEG): container finished" podID="7c1363be-79cd-4103-a588-5a87a285efb1" containerID="40101cf4c486369ba2afdb6addfded141278bfbcbdbb990176bf01f3c29cf99c" exitCode=0
Nov 03 21:25:30.006140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:30.005783    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf" event={"ID":"7c1363be-79cd-4103-a588-5a87a285efb1","Type":"ContainerDied","Data":"40101cf4c486369ba2afdb6addfded141278bfbcbdbb990176bf01f3c29cf99c"}
Nov 03 21:25:31.161821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.161790    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:31.212753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.212728    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\") pod \"7c1363be-79cd-4103-a588-5a87a285efb1\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") "
Nov 03 21:25:31.212903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.212781    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\" (UniqueName: \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\") pod \"7c1363be-79cd-4103-a588-5a87a285efb1\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") "
Nov 03 21:25:31.212903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.212817    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\") pod \"7c1363be-79cd-4103-a588-5a87a285efb1\" (UID: \"7c1363be-79cd-4103-a588-5a87a285efb1\") "
Nov 03 21:25:31.213188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.213167    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca" pod "7c1363be-79cd-4103-a588-5a87a285efb1" (UID: "7c1363be-79cd-4103-a588-5a87a285efb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:31.214728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.214701    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca" pod "7c1363be-79cd-4103-a588-5a87a285efb1" (UID: "7c1363be-79cd-4103-a588-5a87a285efb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:25:31.214812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.214746    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99" pod "7c1363be-79cd-4103-a588-5a87a285efb1" (UID: "7c1363be-79cd-4103-a588-5a87a285efb1"). InnerVolumeSpecName "kube-api-access-89f99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:31.314393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.314324    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:31.314393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.314349    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89f99\" (UniqueName: \"kubernetes.io/projected/7c1363be-79cd-4103-a588-5a87a285efb1-kube-api-access-89f99\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:31.314393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:31.314359    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1363be-79cd-4103-a588-5a87a285efb1-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:32.020580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:32.020549    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"
Nov 03 21:25:32.020580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:32.020559    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf" event={"ID":"7c1363be-79cd-4103-a588-5a87a285efb1","Type":"ContainerDied","Data":"d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5"}
Nov 03 21:25:32.020854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:32.020603    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18f2683d8e08e8a42d8eaf9d496121f9184a6f7546467bf76d375227ebfbbe5"
Nov 03 21:25:35.257979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:35.257945    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:25:36.904987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:36.904952    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:25:37.014003 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:37.013971    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:25:41.905544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:41.905502    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" podUID="b7cae954-b716-4b94-afda-305d058fec58" containerName="payload-processing" containerID="cri-o://6af2675aac0a1ed059a94917e9ba4fa025a36fbb1620dacf5d9f77521561de4c" gracePeriod=25
Nov 03 21:25:42.014626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.014571    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" podUID="3924f959-c529-46ae-ad5b-09dae3d93eb4" containerName="payload-pre-processing" containerID="cri-o://5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1" gracePeriod=25
Nov 03 21:25:42.102199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.101974    5533 generic.go:356] "Generic (PLEG): container finished" podID="b7cae954-b716-4b94-afda-305d058fec58" containerID="6af2675aac0a1ed059a94917e9ba4fa025a36fbb1620dacf5d9f77521561de4c" exitCode=0
Nov 03 21:25:42.102199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.102061    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" event={"ID":"b7cae954-b716-4b94-afda-305d058fec58","Type":"ContainerDied","Data":"6af2675aac0a1ed059a94917e9ba4fa025a36fbb1620dacf5d9f77521561de4c"}
Nov 03 21:25:42.179556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.179522    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:25:42.229683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.228973    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\" (UniqueName: \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\") pod \"b7cae954-b716-4b94-afda-305d058fec58\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") "
Nov 03 21:25:42.229683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.229019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\") pod \"b7cae954-b716-4b94-afda-305d058fec58\" (UID: \"b7cae954-b716-4b94-afda-305d058fec58\") "
Nov 03 21:25:42.230051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.230017    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume" pod "b7cae954-b716-4b94-afda-305d058fec58" (UID: "b7cae954-b716-4b94-afda-305d058fec58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:42.232208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.232179    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l" pod "b7cae954-b716-4b94-afda-305d058fec58" (UID: "b7cae954-b716-4b94-afda-305d058fec58"). InnerVolumeSpecName "kube-api-access-tm96l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:42.279540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.279511    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:25:42.329907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.329880    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\") pod \"3924f959-c529-46ae-ad5b-09dae3d93eb4\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") "
Nov 03 21:25:42.330066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.329941    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\" (UniqueName: \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\") pod \"3924f959-c529-46ae-ad5b-09dae3d93eb4\" (UID: \"3924f959-c529-46ae-ad5b-09dae3d93eb4\") "
Nov 03 21:25:42.330236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.330213    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume" pod "3924f959-c529-46ae-ad5b-09dae3d93eb4" (UID: "3924f959-c529-46ae-ad5b-09dae3d93eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:42.330346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.330332    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3924f959-c529-46ae-ad5b-09dae3d93eb4-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:42.330392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.330348    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm96l\" (UniqueName: \"kubernetes.io/projected/b7cae954-b716-4b94-afda-305d058fec58-kube-api-access-tm96l\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:42.330392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.330357    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cae954-b716-4b94-afda-305d058fec58-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:42.331991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.331971    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb" pod "3924f959-c529-46ae-ad5b-09dae3d93eb4" (UID: "3924f959-c529-46ae-ad5b-09dae3d93eb4"). InnerVolumeSpecName "kube-api-access-ll5gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:42.432090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:42.432014    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll5gb\" (UniqueName: \"kubernetes.io/projected/3924f959-c529-46ae-ad5b-09dae3d93eb4-kube-api-access-ll5gb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:43.109878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.109845    5533 generic.go:356] "Generic (PLEG): container finished" podID="3924f959-c529-46ae-ad5b-09dae3d93eb4" containerID="5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1" exitCode=0
Nov 03 21:25:43.110299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.109909    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"
Nov 03 21:25:43.110299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.109931    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" event={"ID":"3924f959-c529-46ae-ad5b-09dae3d93eb4","Type":"ContainerDied","Data":"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"}
Nov 03 21:25:43.110299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.109987    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx" event={"ID":"3924f959-c529-46ae-ad5b-09dae3d93eb4","Type":"ContainerDied","Data":"c17dc9fa2853c7628ffb1f75633b69de0f0bd0371d0c51334509bb3c6103a96e"}
Nov 03 21:25:43.110299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.110019    5533 scope.go:122] "RemoveContainer" containerID="5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"
Nov 03 21:25:43.111711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.111688    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"
Nov 03 21:25:43.111711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.111695    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt" event={"ID":"b7cae954-b716-4b94-afda-305d058fec58","Type":"ContainerDied","Data":"c1e2d94807a1b5047922a258f495a486391fea9080609f91783e1dce370d1719"}
Nov 03 21:25:43.140244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.134303    5533 scope.go:122] "RemoveContainer" containerID="5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"
Nov 03 21:25:43.140244 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:25:43.136051    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1\": container with ID starting with 5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1 not found: ID does not exist" containerID="5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"
Nov 03 21:25:43.140244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.136098    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1"} err="failed to get container status \"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1\": rpc error: code = NotFound desc = could not find container \"5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1\": container with ID starting with 5c86d17b0b70bf886bb8fcb7e096ea1b31be182fa83078a650861373ec8ae3e1 not found: ID does not exist"
Nov 03 21:25:43.140244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.136146    5533 scope.go:122] "RemoveContainer" containerID="6af2675aac0a1ed059a94917e9ba4fa025a36fbb1620dacf5d9f77521561de4c"
Nov 03 21:25:43.267283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.267207    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:25:43.312268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.312240    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1190cc-5748c666cd-kp4wt"]
Nov 03 21:25:43.371942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.371915    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:25:43.377231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:43.377203    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1190cc-84f4bbf695-pgpbx"]
Nov 03 21:25:44.117347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.117311    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"]
Nov 03 21:25:44.117994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.117975    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b7cae954-b716-4b94-afda-305d058fec58" containerName="payload-processing"
Nov 03 21:25:44.118051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.117997    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cae954-b716-4b94-afda-305d058fec58" containerName="payload-processing"
Nov 03 21:25:44.118051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118023    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3924f959-c529-46ae-ad5b-09dae3d93eb4" containerName="payload-pre-processing"
Nov 03 21:25:44.118051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118028    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3924f959-c529-46ae-ad5b-09dae3d93eb4" containerName="payload-pre-processing"
Nov 03 21:25:44.118051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118035    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7c1363be-79cd-4103-a588-5a87a285efb1" containerName="revoke-keys"
Nov 03 21:25:44.118051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118041    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1363be-79cd-4103-a588-5a87a285efb1" containerName="revoke-keys"
Nov 03 21:25:44.118210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118129    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cae954-b716-4b94-afda-305d058fec58" containerName="payload-processing"
Nov 03 21:25:44.118210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118142    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3924f959-c529-46ae-ad5b-09dae3d93eb4" containerName="payload-pre-processing"
Nov 03 21:25:44.118210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.118152    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1363be-79cd-4103-a588-5a87a285efb1" containerName="revoke-keys"
Nov 03 21:25:44.122664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.122645    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.172420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.172399    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:25:44.175791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.175769    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"]
Nov 03 21:25:44.219285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.219265    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:25:44.252437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.252413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.252556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.252451    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.252658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.252616    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzr2n\" (UniqueName: \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.353325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.353293    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzr2n\" (UniqueName: \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.353456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.353365    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.353456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.353400    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.354006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.353981    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.355799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.355774    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.411029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.410966    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzr2n\" (UniqueName: \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\") pod \"maas-api-revoke-sub-ec227956-xzxj7\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.436047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.436024    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:44.503348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.502398    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3924f959-c529-46ae-ad5b-09dae3d93eb4" path="/var/lib/kubelet/pods/3924f959-c529-46ae-ad5b-09dae3d93eb4/volumes"
Nov 03 21:25:44.503348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.503035    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cae954-b716-4b94-afda-305d058fec58" path="/var/lib/kubelet/pods/b7cae954-b716-4b94-afda-305d058fec58/volumes"
Nov 03 21:25:44.804252 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:25:44.804218    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb024a5b2_e2f6_40a4_afa2_bffbc8b44508.slice/crio-e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83 WatchSource:0}: Error finding container e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83: Status 404 returned error can't find the container with id e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83
Nov 03 21:25:44.804531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:44.804506    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"]
Nov 03 21:25:45.129122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:45.129086    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7" event={"ID":"b024a5b2-e2f6-40a4-afa2-bffbc8b44508","Type":"ContainerStarted","Data":"55bf61a1d8ba79c12ffbaddb2b47bbaedd3050ad10f7bebb521b4762c7c8c720"}
Nov 03 21:25:45.129434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:45.129142    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7" event={"ID":"b024a5b2-e2f6-40a4-afa2-bffbc8b44508","Type":"ContainerStarted","Data":"e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83"}
Nov 03 21:25:45.243793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:45.243692    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7" podStartSLOduration=1.243675812 podStartE2EDuration="1.243675812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:45.242830338 +0000 UTC m=+2246.874143815" watchObservedRunningTime="2026-11-03 21:25:45.243675812 +0000 UTC m=+2246.874989290"
Nov 03 21:25:46.136925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:46.136892    5533 generic.go:356] "Generic (PLEG): container finished" podID="b024a5b2-e2f6-40a4-afa2-bffbc8b44508" containerID="55bf61a1d8ba79c12ffbaddb2b47bbaedd3050ad10f7bebb521b4762c7c8c720" exitCode=0
Nov 03 21:25:46.137336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:46.136981    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7" event={"ID":"b024a5b2-e2f6-40a4-afa2-bffbc8b44508","Type":"ContainerDied","Data":"55bf61a1d8ba79c12ffbaddb2b47bbaedd3050ad10f7bebb521b4762c7c8c720"}
Nov 03 21:25:47.280183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.280159    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:47.382885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.382859    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\") pod \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") "
Nov 03 21:25:47.383029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.382926    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\" (UniqueName: \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\") pod \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") "
Nov 03 21:25:47.383029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.382970    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\") pod \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\" (UID: \"b024a5b2-e2f6-40a4-afa2-bffbc8b44508\") "
Nov 03 21:25:47.383334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.383311    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca" pod "b024a5b2-e2f6-40a4-afa2-bffbc8b44508" (UID: "b024a5b2-e2f6-40a4-afa2-bffbc8b44508"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:25:47.384944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.384918    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca" pod "b024a5b2-e2f6-40a4-afa2-bffbc8b44508" (UID: "b024a5b2-e2f6-40a4-afa2-bffbc8b44508"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:25:47.385047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.384961    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n" pod "b024a5b2-e2f6-40a4-afa2-bffbc8b44508" (UID: "b024a5b2-e2f6-40a4-afa2-bffbc8b44508"). InnerVolumeSpecName "kube-api-access-bzr2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:25:47.484060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.484029    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:47.484060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.484057    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:47.484250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:47.484069    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzr2n\" (UniqueName: \"kubernetes.io/projected/b024a5b2-e2f6-40a4-afa2-bffbc8b44508-kube-api-access-bzr2n\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:25:48.150418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:48.150390    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"
Nov 03 21:25:48.150588 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:48.150431    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7" event={"ID":"b024a5b2-e2f6-40a4-afa2-bffbc8b44508","Type":"ContainerDied","Data":"e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83"}
Nov 03 21:25:48.150588 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:48.150478    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e330cda7b41c5144396c0b1831c29cc1c16bedc30770c69420d995d582974e83"
Nov 03 21:25:49.261987 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:25:49.261949    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:25:51.056055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:51.056022    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"]
Nov 03 21:25:51.104352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:51.104322    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8703dc76-plhcv"]
Nov 03 21:25:52.496051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:25:52.496017    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a550e1e-d442-478d-b1f6-7275066b2067" path="/var/lib/kubelet/pods/6a550e1e-d442-478d-b1f6-7275066b2067/volumes"
Nov 03 21:26:00.075879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.075787    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"]
Nov 03 21:26:00.076363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.076294    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b024a5b2-e2f6-40a4-afa2-bffbc8b44508" containerName="revoke-keys"
Nov 03 21:26:00.076363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.076307    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b024a5b2-e2f6-40a4-afa2-bffbc8b44508" containerName="revoke-keys"
Nov 03 21:26:00.076489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.076471    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b024a5b2-e2f6-40a4-afa2-bffbc8b44508" containerName="revoke-keys"
Nov 03 21:26:00.080985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.080961    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.136976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.136368    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:00.183200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.183177    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:26:00.187308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.187286    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"]
Nov 03 21:26:00.221870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.221840    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.222000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.221874    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqtx\" (UniqueName: \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.222063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.222003    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.323252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.323227    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.323364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.323257    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqtx\" (UniqueName: \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.323364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.323312    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.323900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.323866    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.325719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.325692    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.381127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.381072    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqtx\" (UniqueName: \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\") pod \"maas-api-revoke-sub-881f5c97-cm8vm\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.393888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.393867    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:00.761010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:00.760896    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"]
Nov 03 21:26:00.764042 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:00.764012    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930a1dac_e875_4414_8390_f450a09dabc0.slice/crio-5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2 WatchSource:0}: Error finding container 5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2: Status 404 returned error can't find the container with id 5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2
Nov 03 21:26:01.265689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:01.263807    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm" event={"ID":"930a1dac-e875-4414-8390-f450a09dabc0","Type":"ContainerStarted","Data":"9483d52ab97e1d32a32fe97493983d1b28e17ec4142c661efed7fab050115243"}
Nov 03 21:26:01.265689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:01.264496    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm" event={"ID":"930a1dac-e875-4414-8390-f450a09dabc0","Type":"ContainerStarted","Data":"5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2"}
Nov 03 21:26:01.377560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:01.377470    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm" podStartSLOduration=1.377455893 podStartE2EDuration="1.377455893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:01.374726771 +0000 UTC m=+2263.006040247" watchObservedRunningTime="2026-11-03 21:26:01.377455893 +0000 UTC m=+2263.008769394"
Nov 03 21:26:02.176913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.176875    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:26:02.181172 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.181145    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.242101    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2285b9-openshift-default-dockercfg-nf25m\""
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262469    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262515    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262558    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262592    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262673    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262856    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262888    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262918    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.266213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.262974    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nrk\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.285500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.284196    5533 generic.go:356] "Generic (PLEG): container finished" podID="930a1dac-e875-4414-8390-f450a09dabc0" containerID="9483d52ab97e1d32a32fe97493983d1b28e17ec4142c661efed7fab050115243" exitCode=0
Nov 03 21:26:02.285500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.284424    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm" event={"ID":"930a1dac-e875-4414-8390-f450a09dabc0","Type":"ContainerDied","Data":"9483d52ab97e1d32a32fe97493983d1b28e17ec4142c661efed7fab050115243"}
Nov 03 21:26:02.299251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.299019    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363616    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363736    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363782    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nrk\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363851    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.363951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.363877    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.367686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.364670    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.367686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.365742    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.367686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.366552    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.368555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.368198    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.368555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.368257    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.368555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.368331    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.372153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.370886    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.372153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.371138    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.374676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.374646    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.375154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.375120    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.424795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.424763    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.424951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.424937    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nrk\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\") pod \"e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") " pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.494081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.494008    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:02.913527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.913495    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:26:02.914075 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:02.914050    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bc06c5_3d49_47d9_ba9d_9154b10e0c98.slice/crio-4da106620fd155cb021812a9642c711c9f5cda94c51b4d6e662de608a0a5cd28 WatchSource:0}: Error finding container 4da106620fd155cb021812a9642c711c9f5cda94c51b4d6e662de608a0a5cd28: Status 404 returned error can't find the container with id 4da106620fd155cb021812a9642c711c9f5cda94c51b4d6e662de608a0a5cd28
Nov 03 21:26:02.916731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.916697    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:02.916845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.916779    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:02.916845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:02.916808    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:03.292556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.292521    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerStarted","Data":"d96cbf24862bf486c02f2d1c57c3f44b320f024c188cc15f29c04685ca70feaf"}
Nov 03 21:26:03.293025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.292583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerStarted","Data":"4da106620fd155cb021812a9642c711c9f5cda94c51b4d6e662de608a0a5cd28"}
Nov 03 21:26:03.407085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.407026    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podStartSLOduration=1.407009489 podStartE2EDuration="1.407009489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:03.405247488 +0000 UTC m=+2265.036560965" watchObservedRunningTime="2026-11-03 21:26:03.407009489 +0000 UTC m=+2265.038322966"
Nov 03 21:26:03.442141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.442114    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:03.482105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.482079    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\" (UniqueName: \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\") pod \"930a1dac-e875-4414-8390-f450a09dabc0\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") "
Nov 03 21:26:03.482244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.482226    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\") pod \"930a1dac-e875-4414-8390-f450a09dabc0\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") "
Nov 03 21:26:03.482299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.482261    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\") pod \"930a1dac-e875-4414-8390-f450a09dabc0\" (UID: \"930a1dac-e875-4414-8390-f450a09dabc0\") "
Nov 03 21:26:03.482560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.482535    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca" pod "930a1dac-e875-4414-8390-f450a09dabc0" (UID: "930a1dac-e875-4414-8390-f450a09dabc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:03.484205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.484178    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx" pod "930a1dac-e875-4414-8390-f450a09dabc0" (UID: "930a1dac-e875-4414-8390-f450a09dabc0"). InnerVolumeSpecName "kube-api-access-9dqtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:03.484292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.484256    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca" pod "930a1dac-e875-4414-8390-f450a09dabc0" (UID: "930a1dac-e875-4414-8390-f450a09dabc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:03.494870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.494849    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:03.501294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.501275    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:03.583788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.583579    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:03.583788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.583650    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930a1dac-e875-4414-8390-f450a09dabc0-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:03.583788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:03.583679    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqtx\" (UniqueName: \"kubernetes.io/projected/930a1dac-e875-4414-8390-f450a09dabc0-kube-api-access-9dqtx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:04.300737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:04.300706    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"
Nov 03 21:26:04.301119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:04.300746    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm" event={"ID":"930a1dac-e875-4414-8390-f450a09dabc0","Type":"ContainerDied","Data":"5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2"}
Nov 03 21:26:04.301119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:04.300793    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5121553faa534b33c8e8f7f45393fe0ea563021cfd743c417c2e6189a38ac9e2"
Nov 03 21:26:04.301119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:04.301090    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:04.302030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:04.302008    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:26:07.288158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.288124    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:26:07.288650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.288620    5533 cpu_manager.go:414] "containerMap: removing container" podUID="930a1dac-e875-4414-8390-f450a09dabc0" containerName="revoke-keys"
Nov 03 21:26:07.288650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.288643    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930a1dac-e875-4414-8390-f450a09dabc0" containerName="revoke-keys"
Nov 03 21:26:07.288785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.288765    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="930a1dac-e875-4414-8390-f450a09dabc0" containerName="revoke-keys"
Nov 03 21:26:07.292363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.292339    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.329913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.329883    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.329924    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330004    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6t8\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330107    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330145    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330273    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330334    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330374    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.330404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.330394    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.341434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.341408    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e84357-openshift-default-dockercfg-t4w57\""
Nov 03 21:26:07.393143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.393112    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:26:07.431381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431345    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.431568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431393    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.431568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431455    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.431568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431483    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.431568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431502    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.431568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431520    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431892    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.431997    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432043    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432182 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432087    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6t8\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432269    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432500    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432697    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.432899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.432870    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.434532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.434503    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.434792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.434768    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.489683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.489658    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.489813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.489791    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6t8\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:07.605616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:07.605548    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:08.027486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.027448    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:26:08.028964 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:08.028934    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b44d7b_7466_44c0_a4db_84ff6b66a21b.slice/crio-bb193d2bdfe4ef59a0b7b8c9af3e9be1e32ac90d6f1a38459200da9ba722dea0 WatchSource:0}: Error finding container bb193d2bdfe4ef59a0b7b8c9af3e9be1e32ac90d6f1a38459200da9ba722dea0: Status 404 returned error can't find the container with id bb193d2bdfe4ef59a0b7b8c9af3e9be1e32ac90d6f1a38459200da9ba722dea0
Nov 03 21:26:08.031575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.031535    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:08.031731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.031623    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:08.031731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.031693    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:26:08.342297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.342200    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" event={"ID":"74b44d7b-7466-44c0-a4db-84ff6b66a21b","Type":"ContainerStarted","Data":"e78420ad35a3831fbc2cb642b1a0f748cf891fa4eacfae7807731b8a21979dd0"}
Nov 03 21:26:08.342297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.342280    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" event={"ID":"74b44d7b-7466-44c0-a4db-84ff6b66a21b","Type":"ContainerStarted","Data":"bb193d2bdfe4ef59a0b7b8c9af3e9be1e32ac90d6f1a38459200da9ba722dea0"}
Nov 03 21:26:08.459945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.459890    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podStartSLOduration=1.459875396 podStartE2EDuration="1.459875396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:08.4560814 +0000 UTC m=+2270.087394878" watchObservedRunningTime="2026-11-03 21:26:08.459875396 +0000 UTC m=+2270.091188934"
Nov 03 21:26:08.606814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:08.606722    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:09.633672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:09.631789    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:09.633672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:09.631855    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:09.998997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:09.997856    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:26:10.036658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.036010    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.073685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.069804    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:26:10.113658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.103915    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2285b9-dockercfg-xb4k8\""
Nov 03 21:26:10.113658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.104029    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2285b9\""
Nov 03 21:26:10.120354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.120319    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqr2\" (UniqueName: \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.120514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.120443    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.183500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.183473    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:26:10.187893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.187871    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.222122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.222096    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqr2\" (UniqueName: \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.222253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.222146    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.222795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.222776    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.242968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.242942    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:26:10.284617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.284547    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqr2\" (UniqueName: \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\") pod \"payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.323694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.323660    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.323882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.323818    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfhw\" (UniqueName: \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.425273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.425241    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.425470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.425451    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfhw\" (UniqueName: \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.425898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.425879    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.430041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.430023    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:10.489273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.489207    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfhw\" (UniqueName: \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\") pod \"payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.504420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.504207    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:10.606671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.606556    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:10.606671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.606609    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:10.837675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.837649    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:26:10.839072 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:10.839033    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06ff874_c3ef_439d_91d4_ab9dfdddb3b2.slice/crio-2a29edb9f91eb1d645083a165c7c2f3f2347428ba0e449a52e80a353ac802832 WatchSource:0}: Error finding container 2a29edb9f91eb1d645083a165c7c2f3f2347428ba0e449a52e80a353ac802832: Status 404 returned error can't find the container with id 2a29edb9f91eb1d645083a165c7c2f3f2347428ba0e449a52e80a353ac802832
Nov 03 21:26:10.937691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:10.936793    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:26:10.943659 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:10.943576    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde213a03_d758_4007_a2d3_5777d052377b.slice/crio-84a7308eb977a2919ce5e2ab42e76e47f8a05694dc4e25bef8fe321d6e36028e WatchSource:0}: Error finding container 84a7308eb977a2919ce5e2ab42e76e47f8a05694dc4e25bef8fe321d6e36028e: Status 404 returned error can't find the container with id 84a7308eb977a2919ce5e2ab42e76e47f8a05694dc4e25bef8fe321d6e36028e
Nov 03 21:26:11.387928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.387884    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" event={"ID":"de213a03-d758-4007-a2d3-5777d052377b","Type":"ContainerStarted","Data":"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"}
Nov 03 21:26:11.388116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.388089    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:11.388184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.388125    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" event={"ID":"de213a03-d758-4007-a2d3-5777d052377b","Type":"ContainerStarted","Data":"84a7308eb977a2919ce5e2ab42e76e47f8a05694dc4e25bef8fe321d6e36028e"}
Nov 03 21:26:11.389572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.389539    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" event={"ID":"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2","Type":"ContainerStarted","Data":"5533f9f7f676b120951a59619df1eae93c3fd785cd78465a80c565693da4d4cb"}
Nov 03 21:26:11.389792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.389582    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" event={"ID":"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2","Type":"ContainerStarted","Data":"2a29edb9f91eb1d645083a165c7c2f3f2347428ba0e449a52e80a353ac802832"}
Nov 03 21:26:11.389792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.389607    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:11.606598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.606145    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:11.606598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.606208    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:11.614088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.614036    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" podStartSLOduration=1.614016974 podStartE2EDuration="1.614016974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:11.500875863 +0000 UTC m=+2273.132189340" watchObservedRunningTime="2026-11-03 21:26:11.614016974 +0000 UTC m=+2273.245330452"
Nov 03 21:26:11.617068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:11.617025    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" podStartSLOduration=2.617011061 podStartE2EDuration="2.617011061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:11.612818754 +0000 UTC m=+2273.244132230" watchObservedRunningTime="2026-11-03 21:26:11.617011061 +0000 UTC m=+2273.248324541"
Nov 03 21:26:12.606568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:12.606528    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:12.607048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:12.606579    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:13.606294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:13.606261    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:13.606490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:13.606312    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:14.606379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:14.606346    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 03 21:26:14.606781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:14.606396    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 03 21:26:15.611302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:15.611274    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:15.611827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:15.611676    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:15.612308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:15.612290    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:26:17.257136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.257094    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:26:17.257665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.257345    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" containerName="discovery" containerID="cri-o://b4af9e549d4e48d008b3e1ca67b756664f5df460de0d4647f84ee7399b6a5dd7" gracePeriod=30
Nov 03 21:26:17.302093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.302064    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"]
Nov 03 21:26:17.306180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.306161    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.357701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.357675    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:17.357977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.357843    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:26:17.361833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.361806    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"]
Nov 03 21:26:17.433616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.433581    5533 generic.go:356] "Generic (PLEG): container finished" podID="14d18783-a004-48bb-b664-ef1edf7ea720" containerID="b4af9e549d4e48d008b3e1ca67b756664f5df460de0d4647f84ee7399b6a5dd7" exitCode=0
Nov 03 21:26:17.434602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.434578    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" event={"ID":"14d18783-a004-48bb-b664-ef1edf7ea720","Type":"ContainerDied","Data":"b4af9e549d4e48d008b3e1ca67b756664f5df460de0d4647f84ee7399b6a5dd7"}
Nov 03 21:26:17.435327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.435302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sks4t\" (UniqueName: \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.435520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.435497    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.435572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.435540    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.536399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.536369    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sks4t\" (UniqueName: \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.536737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.536695    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.536838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.536750    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.537493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.537465    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.539033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.539010    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.554864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.554844    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:26:17.594232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.594203    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sks4t\" (UniqueName: \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\") pod \"maas-api-revoke-sub-f06bfca6-c494c\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.637333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637309    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637369    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637469    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637525    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637620    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637685    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637755    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\") pod \"14d18783-a004-48bb-b664-ef1edf7ea720\" (UID: \"14d18783-a004-48bb-b664-ef1edf7ea720\") "
Nov 03 21:26:17.637916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.637889    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:17.638323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.638297    5533 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-ca-configmap\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.639810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.639791    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:17.640623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.640593    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "kube-api-access-zl4gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:17.640757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.640737    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:17.640840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.640807    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:17.641231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.641216    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:17.643065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.643032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts" pod "14d18783-a004-48bb-b664-ef1edf7ea720" (UID: "14d18783-a004-48bb-b664-ef1edf7ea720"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:17.663231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.663200    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743818    5533 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-kubeconfig\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743855    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743870    5533 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-istio-csr-dns-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743885    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4gx\" (UniqueName: \"kubernetes.io/projected/14d18783-a004-48bb-b664-ef1edf7ea720-kube-api-access-zl4gx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743901    5533 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d18783-a004-48bb-b664-ef1edf7ea720-cacerts\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:17.743942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:17.743916    5533 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d18783-a004-48bb-b664-ef1edf7ea720-local-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:18.048492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.048436    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"]
Nov 03 21:26:18.051538 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:18.051503    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189da769_4335_4ecc_a108_2a355ed33e5f.slice/crio-278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1 WatchSource:0}: Error finding container 278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1: Status 404 returned error can't find the container with id 278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1
Nov 03 21:26:18.449530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.449490    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c" event={"ID":"189da769-4335-4ecc-a108-2a355ed33e5f","Type":"ContainerStarted","Data":"77bd309f4f2a5380e1603da54b24bdfdda9c84216b63462795364f450a89d1ab"}
Nov 03 21:26:18.449946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.449580    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c" event={"ID":"189da769-4335-4ecc-a108-2a355ed33e5f","Type":"ContainerStarted","Data":"278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1"}
Nov 03 21:26:18.451042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.451017    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"
Nov 03 21:26:18.451042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.451038    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj" event={"ID":"14d18783-a004-48bb-b664-ef1edf7ea720","Type":"ContainerDied","Data":"43da674823121ce56dff2aae733bd033bc0c73f6cd244a78c6935e2b6938c52e"}
Nov 03 21:26:18.451242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.451103    5533 scope.go:122] "RemoveContainer" containerID="b4af9e549d4e48d008b3e1ca67b756664f5df460de0d4647f84ee7399b6a5dd7"
Nov 03 21:26:18.559205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.559149    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c" podStartSLOduration=1.559133116 podStartE2EDuration="1.559133116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:18.557788784 +0000 UTC m=+2280.189102261" watchObservedRunningTime="2026-11-03 21:26:18.559133116 +0000 UTC m=+2280.190446594"
Nov 03 21:26:18.718426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.718399    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:26:18.724130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:18.724106    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6d5tj"]
Nov 03 21:26:19.262253 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:26:19.262216    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:26:19.460061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:19.460025    5533 generic.go:356] "Generic (PLEG): container finished" podID="189da769-4335-4ecc-a108-2a355ed33e5f" containerID="77bd309f4f2a5380e1603da54b24bdfdda9c84216b63462795364f450a89d1ab" exitCode=0
Nov 03 21:26:19.460514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:19.460120    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c" event={"ID":"189da769-4335-4ecc-a108-2a355ed33e5f","Type":"ContainerDied","Data":"77bd309f4f2a5380e1603da54b24bdfdda9c84216b63462795364f450a89d1ab"}
Nov 03 21:26:20.497030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.496994    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" path="/var/lib/kubelet/pods/14d18783-a004-48bb-b664-ef1edf7ea720/volumes"
Nov 03 21:26:20.623371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.623347    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:20.780037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.779958    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\" (UniqueName: \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\") pod \"189da769-4335-4ecc-a108-2a355ed33e5f\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") "
Nov 03 21:26:20.780186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.780153    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\") pod \"189da769-4335-4ecc-a108-2a355ed33e5f\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") "
Nov 03 21:26:20.780259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.780192    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\") pod \"189da769-4335-4ecc-a108-2a355ed33e5f\" (UID: \"189da769-4335-4ecc-a108-2a355ed33e5f\") "
Nov 03 21:26:20.780651 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.780606    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca" pod "189da769-4335-4ecc-a108-2a355ed33e5f" (UID: "189da769-4335-4ecc-a108-2a355ed33e5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:20.782183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.782154    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t" pod "189da769-4335-4ecc-a108-2a355ed33e5f" (UID: "189da769-4335-4ecc-a108-2a355ed33e5f"). InnerVolumeSpecName "kube-api-access-sks4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:20.782607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.782587    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca" pod "189da769-4335-4ecc-a108-2a355ed33e5f" (UID: "189da769-4335-4ecc-a108-2a355ed33e5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:20.882150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.882125    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:20.882267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.882168    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/189da769-4335-4ecc-a108-2a355ed33e5f-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:20.882267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:20.882178    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sks4t\" (UniqueName: \"kubernetes.io/projected/189da769-4335-4ecc-a108-2a355ed33e5f-kube-api-access-sks4t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:21.480422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:21.480392    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"
Nov 03 21:26:21.480422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:21.480402    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c" event={"ID":"189da769-4335-4ecc-a108-2a355ed33e5f","Type":"ContainerDied","Data":"278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1"}
Nov 03 21:26:21.480621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:21.480473    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278d52b98cd4d76158ff5e62063fca4022c5b37fb9456f1622ba31f9c08b00e1"
Nov 03 21:26:22.053838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:22.053798    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:26:22.400650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:22.400565    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:26:22.400790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:22.400664    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:26:24.494743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:24.494710    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:26:24.599319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:24.599288    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:26:25.737856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:25.737820    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:26:25.738315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:25.738006    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" containerID="cri-o://28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d" gracePeriod=30
Nov 03 21:26:27.209831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.209800    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"]
Nov 03 21:26:27.210420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210402    5533 cpu_manager.go:414] "containerMap: removing container" podUID="189da769-4335-4ecc-a108-2a355ed33e5f" containerName="revoke-keys"
Nov 03 21:26:27.210499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210421    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189da769-4335-4ecc-a108-2a355ed33e5f" containerName="revoke-keys"
Nov 03 21:26:27.210499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210452    5533 cpu_manager.go:414] "containerMap: removing container" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" containerName="discovery"
Nov 03 21:26:27.210499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210458    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" containerName="discovery"
Nov 03 21:26:27.210630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210538    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d18783-a004-48bb-b664-ef1edf7ea720" containerName="discovery"
Nov 03 21:26:27.210630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.210550    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="189da769-4335-4ecc-a108-2a355ed33e5f" containerName="revoke-keys"
Nov 03 21:26:27.214736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.214716    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.271453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.271426    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"]
Nov 03 21:26:27.271923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.271900    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.272033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.271956    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.272097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.272036    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrztv\" (UniqueName: \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.312727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.312701    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:27.312843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.312752    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:26:27.373235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.373205    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.373369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.373298    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.373369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.373336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrztv\" (UniqueName: \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.373911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.373889    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.376326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.376299    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.430988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.430960    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrztv\" (UniqueName: \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\") pod \"maas-api-revoke-sub-b28f25e5-l9vz2\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.537893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.537830    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:27.909542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:27.909504    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"]
Nov 03 21:26:27.910512 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:27.910479    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c083a0_c08a_48e4_8863_15c1c57e0f34.slice/crio-b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675 WatchSource:0}: Error finding container b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675: Status 404 returned error can't find the container with id b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675
Nov 03 21:26:28.070089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.070060    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"]
Nov 03 21:26:28.085383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.085240    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.128595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.128567    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"]
Nov 03 21:26:28.184355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.184327    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.184505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.184409    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hv7\" (UniqueName: \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.184505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.184431    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.285902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.285870    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.286394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.285984    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hv7\" (UniqueName: \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.286394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.286016    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.286793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.286767    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.290014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.288286    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.343501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.343472    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hv7\" (UniqueName: \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\") pod \"maas-api-revoke-sub-3bdb9fd6-tlz4g\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.398957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.398927    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:28.531549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.531509    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2" event={"ID":"c1c083a0-c08a-48e4-8863-15c1c57e0f34","Type":"ContainerStarted","Data":"706935f9c1a523ce6e93c0b5e21730b8fc91801c3e86502d0a888242e49ec8b3"}
Nov 03 21:26:28.531748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.531579    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2" event={"ID":"c1c083a0-c08a-48e4-8863-15c1c57e0f34","Type":"ContainerStarted","Data":"b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675"}
Nov 03 21:26:28.641274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.641217    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2" podStartSLOduration=1.641204597 podStartE2EDuration="1.641204597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:28.640206843 +0000 UTC m=+2290.271520319" watchObservedRunningTime="2026-11-03 21:26:28.641204597 +0000 UTC m=+2290.272518074"
Nov 03 21:26:28.779210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.779180    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"]
Nov 03 21:26:28.779776 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:28.779661    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d40a61e_6aac_4691_9dc2_228902421d97.slice/crio-2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b WatchSource:0}: Error finding container 2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b: Status 404 returned error can't find the container with id 2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b
Nov 03 21:26:28.805884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.805847    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:26:28.805993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:28.805902    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:26:29.496066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.496027    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" podUID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" containerName="payload-processing" containerID="cri-o://749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb" gracePeriod=25
Nov 03 21:26:29.539405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.539371    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g" event={"ID":"9d40a61e-6aac-4691-9dc2-228902421d97","Type":"ContainerStarted","Data":"8fb3f6421cd525cc5fb425a393d364e18f9e4598eb71da05396a8229e828a2b2"}
Nov 03 21:26:29.539537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.539434    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g" event={"ID":"9d40a61e-6aac-4691-9dc2-228902421d97","Type":"ContainerStarted","Data":"2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b"}
Nov 03 21:26:29.541186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.541161    5533 generic.go:356] "Generic (PLEG): container finished" podID="c1c083a0-c08a-48e4-8863-15c1c57e0f34" containerID="706935f9c1a523ce6e93c0b5e21730b8fc91801c3e86502d0a888242e49ec8b3" exitCode=0
Nov 03 21:26:29.541282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.541208    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2" event={"ID":"c1c083a0-c08a-48e4-8863-15c1c57e0f34","Type":"ContainerDied","Data":"706935f9c1a523ce6e93c0b5e21730b8fc91801c3e86502d0a888242e49ec8b3"}
Nov 03 21:26:29.599580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.599527    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" podUID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" containerName="payload-pre-processing" containerID="cri-o://0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65" gracePeriod=25
Nov 03 21:26:29.649104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.649017    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g" podStartSLOduration=1.649004186 podStartE2EDuration="1.649004186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:29.647546184 +0000 UTC m=+2291.278859673" watchObservedRunningTime="2026-11-03 21:26:29.649004186 +0000 UTC m=+2291.280317692"
Nov 03 21:26:29.796666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.796511    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:26:29.850900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.850876    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:26:29.906155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906085    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\" (UniqueName: \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\") pod \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") "
Nov 03 21:26:29.906294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906236    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\" (UniqueName: \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\") pod \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") "
Nov 03 21:26:29.906294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906284    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\") pod \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\" (UID: \"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc\") "
Nov 03 21:26:29.906471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906353    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\") pod \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\" (UID: \"57a372f5-8b6d-4dba-ae9c-63c6b528e82e\") "
Nov 03 21:26:29.906847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906741    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume" pod "57a372f5-8b6d-4dba-ae9c-63c6b528e82e" (UID: "57a372f5-8b6d-4dba-ae9c-63c6b528e82e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:29.906847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.906793    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume" pod "0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" (UID: "0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:29.908885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.908857    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz" pod "0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" (UID: "0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc"). InnerVolumeSpecName "kube-api-access-f2cgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:29.908885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:29.908884    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht" pod "57a372f5-8b6d-4dba-ae9c-63c6b528e82e" (UID: "57a372f5-8b6d-4dba-ae9c-63c6b528e82e"). InnerVolumeSpecName "kube-api-access-pwtht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:30.008264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.008231    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.008264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.008258    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cgz\" (UniqueName: \"kubernetes.io/projected/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-kube-api-access-f2cgz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.008264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.008269    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtht\" (UniqueName: \"kubernetes.io/projected/57a372f5-8b6d-4dba-ae9c-63c6b528e82e-kube-api-access-pwtht\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.008493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.008278    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.548330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.548298    5533 generic.go:356] "Generic (PLEG): container finished" podID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" containerID="749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb" exitCode=0
Nov 03 21:26:30.548718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.548346    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"
Nov 03 21:26:30.548718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.548369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" event={"ID":"57a372f5-8b6d-4dba-ae9c-63c6b528e82e","Type":"ContainerDied","Data":"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"}
Nov 03 21:26:30.548718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.548417    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw" event={"ID":"57a372f5-8b6d-4dba-ae9c-63c6b528e82e","Type":"ContainerDied","Data":"19a261c94d88f4c72ac7e7ed5ee001b2cd2aec2f6c8e32c9e98d85856e4f2633"}
Nov 03 21:26:30.548718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.548449    5533 scope.go:122] "RemoveContainer" containerID="749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"
Nov 03 21:26:30.550386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.550360    5533 generic.go:356] "Generic (PLEG): container finished" podID="9d40a61e-6aac-4691-9dc2-228902421d97" containerID="8fb3f6421cd525cc5fb425a393d364e18f9e4598eb71da05396a8229e828a2b2" exitCode=0
Nov 03 21:26:30.550494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.550437    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g" event={"ID":"9d40a61e-6aac-4691-9dc2-228902421d97","Type":"ContainerDied","Data":"8fb3f6421cd525cc5fb425a393d364e18f9e4598eb71da05396a8229e828a2b2"}
Nov 03 21:26:30.552204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.552183    5533 generic.go:356] "Generic (PLEG): container finished" podID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" containerID="0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65" exitCode=0
Nov 03 21:26:30.552315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.552237    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"
Nov 03 21:26:30.552315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.552287    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" event={"ID":"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc","Type":"ContainerDied","Data":"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"}
Nov 03 21:26:30.552444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.552325    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5" event={"ID":"0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc","Type":"ContainerDied","Data":"5b27a4776aee1fe02c1312e4d3757a3bfb22b1b974feb67a1c5257a15d537297"}
Nov 03 21:26:30.561324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.561304    5533 scope.go:122] "RemoveContainer" containerID="749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"
Nov 03 21:26:30.561606 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:26:30.561585    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb\": container with ID starting with 749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb not found: ID does not exist" containerID="749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"
Nov 03 21:26:30.562061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.561610    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb"} err="failed to get container status \"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb\": rpc error: code = NotFound desc = could not find container \"749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb\": container with ID starting with 749beca376026288bb626798067cb7827d94a4d2a1518f23a0f47367cd8821eb not found: ID does not exist"
Nov 03 21:26:30.562061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.561648    5533 scope.go:122] "RemoveContainer" containerID="0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"
Nov 03 21:26:30.573588 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.573569    5533 scope.go:122] "RemoveContainer" containerID="0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"
Nov 03 21:26:30.573909 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:26:30.573882    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65\": container with ID starting with 0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65 not found: ID does not exist" containerID="0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"
Nov 03 21:26:30.573983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.573920    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65"} err="failed to get container status \"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65\": rpc error: code = NotFound desc = could not find container \"0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65\": container with ID starting with 0840bd5c1f6f3fa560d9cd713b254281f1d350e467921c3a86586f7c849dba65 not found: ID does not exist"
Nov 03 21:26:30.706118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.706086    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:30.813029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.813000    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:26:30.816752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.816728    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1dda2c-ccf999b7-2tgg5"]
Nov 03 21:26:30.819027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.819002    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\") pod \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") "
Nov 03 21:26:30.819156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.819099    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\" (UniqueName: \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\") pod \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") "
Nov 03 21:26:30.819229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.819161    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\") pod \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\" (UID: \"c1c083a0-c08a-48e4-8863-15c1c57e0f34\") "
Nov 03 21:26:30.819554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.819531    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca" pod "c1c083a0-c08a-48e4-8863-15c1c57e0f34" (UID: "c1c083a0-c08a-48e4-8863-15c1c57e0f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:30.821358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.821335    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv" pod "c1c083a0-c08a-48e4-8863-15c1c57e0f34" (UID: "c1c083a0-c08a-48e4-8863-15c1c57e0f34"). InnerVolumeSpecName "kube-api-access-vrztv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:30.835921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.835892    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca" pod "c1c083a0-c08a-48e4-8863-15c1c57e0f34" (UID: "c1c083a0-c08a-48e4-8863-15c1c57e0f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:30.921681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.921486    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.921837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.921691    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrztv\" (UniqueName: \"kubernetes.io/projected/c1c083a0-c08a-48e4-8863-15c1c57e0f34-kube-api-access-vrztv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.921837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.921711    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c083a0-c08a-48e4-8863-15c1c57e0f34-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:30.925501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.925467    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:26:30.936090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.935988    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1dda2c-7fb8585fb4-c24zw"]
Nov 03 21:26:30.992706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:30.992679    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:26:31.122893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.122860    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.122936    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123002    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123056    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123096    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123125    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123161    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123252    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:31.123432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\") pod \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\" (UID: \"f3758ae8-f613-46fd-9e22-8497c43c9e1a\") "
Nov 03 21:26:31.123432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123379    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:31.123572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123431    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:31.123572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123467    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:31.123572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123496    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:31.123853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123832    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.123955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123857    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.123955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123872    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.123955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123885    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.123955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.123898    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.125074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.125052    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:26:31.125460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.125444    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:31.125649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.125616    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:26:31.125718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.125626    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j" pod "f3758ae8-f613-46fd-9e22-8497c43c9e1a" (UID: "f3758ae8-f613-46fd-9e22-8497c43c9e1a"). InnerVolumeSpecName "kube-api-access-cn64j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:31.225123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.225047    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.225123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.225076    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn64j\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-kube-api-access-cn64j\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.225123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.225086    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.225123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.225096    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3758ae8-f613-46fd-9e22-8497c43c9e1a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.562360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.562274    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2" event={"ID":"c1c083a0-c08a-48e4-8863-15c1c57e0f34","Type":"ContainerDied","Data":"b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675"}
Nov 03 21:26:31.562360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.562339    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b34acc0213820b954c4fc0eb210d00b9eeb4fd6c28038450d49220359dd675"
Nov 03 21:26:31.562360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.562296    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"
Nov 03 21:26:31.563882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.563856    5533 generic.go:356] "Generic (PLEG): container finished" podID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerID="28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d" exitCode=0
Nov 03 21:26:31.564032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.563896    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"
Nov 03 21:26:31.564032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.563940    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" event={"ID":"f3758ae8-f613-46fd-9e22-8497c43c9e1a","Type":"ContainerDied","Data":"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"}
Nov 03 21:26:31.564032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.563991    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn" event={"ID":"f3758ae8-f613-46fd-9e22-8497c43c9e1a","Type":"ContainerDied","Data":"920b529bf3f86e2f9a64a5dbf46a0283242486e0f222504919a8ba810f83d90c"}
Nov 03 21:26:31.564032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.564014    5533 scope.go:122] "RemoveContainer" containerID="28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"
Nov 03 21:26:31.576424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.576405    5533 scope.go:122] "RemoveContainer" containerID="28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"
Nov 03 21:26:31.576714 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:26:31.576694    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d\": container with ID starting with 28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d not found: ID does not exist" containerID="28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"
Nov 03 21:26:31.576812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.576721    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d"} err="failed to get container status \"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d\": rpc error: code = NotFound desc = could not find container \"28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d\": container with ID starting with 28511c52103d14f64bed52571e115f59cf68f8e216033fb33c697b91f49d604d not found: ID does not exist"
Nov 03 21:26:31.723367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.723339    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:26:31.725437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.725413    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:31.729492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.729472    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-nxsrn"]
Nov 03 21:26:31.830084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.830024    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\") pod \"9d40a61e-6aac-4691-9dc2-228902421d97\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") "
Nov 03 21:26:31.830084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.830054    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\" (UniqueName: \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\") pod \"9d40a61e-6aac-4691-9dc2-228902421d97\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") "
Nov 03 21:26:31.830216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.830204    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\") pod \"9d40a61e-6aac-4691-9dc2-228902421d97\" (UID: \"9d40a61e-6aac-4691-9dc2-228902421d97\") "
Nov 03 21:26:31.830527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.830493    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca" pod "9d40a61e-6aac-4691-9dc2-228902421d97" (UID: "9d40a61e-6aac-4691-9dc2-228902421d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:31.830696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.830677    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.832155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.832129    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca" pod "9d40a61e-6aac-4691-9dc2-228902421d97" (UID: "9d40a61e-6aac-4691-9dc2-228902421d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:31.832155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.832150    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7" pod "9d40a61e-6aac-4691-9dc2-228902421d97" (UID: "9d40a61e-6aac-4691-9dc2-228902421d97"). InnerVolumeSpecName "kube-api-access-n9hv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:31.931398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.931369    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d40a61e-6aac-4691-9dc2-228902421d97-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:31.931398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:31.931396    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9hv7\" (UniqueName: \"kubernetes.io/projected/9d40a61e-6aac-4691-9dc2-228902421d97-kube-api-access-n9hv7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:32.493502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.493461    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" path="/var/lib/kubelet/pods/0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc/volumes"
Nov 03 21:26:32.493928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.493912    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" path="/var/lib/kubelet/pods/57a372f5-8b6d-4dba-ae9c-63c6b528e82e/volumes"
Nov 03 21:26:32.494365 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.494348    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" path="/var/lib/kubelet/pods/f3758ae8-f613-46fd-9e22-8497c43c9e1a/volumes"
Nov 03 21:26:32.576381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.576358    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"
Nov 03 21:26:32.576381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.576373    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g" event={"ID":"9d40a61e-6aac-4691-9dc2-228902421d97","Type":"ContainerDied","Data":"2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b"}
Nov 03 21:26:32.576759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:32.576426    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2809b25a3bf9d73a630d748037d4ced95a3f8959da5ce2e8baf44a162f54639b"
Nov 03 21:26:44.252657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.252605    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"]
Nov 03 21:26:44.253325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253303    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9d40a61e-6aac-4691-9dc2-228902421d97" containerName="revoke-keys"
Nov 03 21:26:44.253377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253327    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d40a61e-6aac-4691-9dc2-228902421d97" containerName="revoke-keys"
Nov 03 21:26:44.253377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253354    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" containerName="payload-pre-processing"
Nov 03 21:26:44.253377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253360    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" containerName="payload-pre-processing"
Nov 03 21:26:44.253377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253367    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c1c083a0-c08a-48e4-8863-15c1c57e0f34" containerName="revoke-keys"
Nov 03 21:26:44.253377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253372    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c083a0-c08a-48e4-8863-15c1c57e0f34" containerName="revoke-keys"
Nov 03 21:26:44.253527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253402    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy"
Nov 03 21:26:44.253527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253407    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy"
Nov 03 21:26:44.253527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253419    5533 cpu_manager.go:414] "containerMap: removing container" podUID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" containerName="payload-processing"
Nov 03 21:26:44.253527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253425    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" containerName="payload-processing"
Nov 03 21:26:44.253711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253525    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0e87f7-e4cf-4f57-a9f3-7e870c3d0bdc" containerName="payload-pre-processing"
Nov 03 21:26:44.253711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253536    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c083a0-c08a-48e4-8863-15c1c57e0f34" containerName="revoke-keys"
Nov 03 21:26:44.253711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253554    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3758ae8-f613-46fd-9e22-8497c43c9e1a" containerName="istio-proxy"
Nov 03 21:26:44.253711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253564    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a372f5-8b6d-4dba-ae9c-63c6b528e82e" containerName="payload-processing"
Nov 03 21:26:44.253711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.253575    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d40a61e-6aac-4691-9dc2-228902421d97" containerName="revoke-keys"
Nov 03 21:26:44.256875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.256855    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.306795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.306764    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:44.310769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.310740    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"]
Nov 03 21:26:44.353507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.353488    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:26:44.358141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.358116    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddg92\" (UniqueName: \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.358221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.358169    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.358366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.358336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.459096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.459064    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.459236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.459158    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.459337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.459278    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddg92\" (UniqueName: \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.459866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.459846    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.461647 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.461618    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.517048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.516995    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddg92\" (UniqueName: \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\") pod \"maas-api-revoke-sub-86d35cae-xrlx7\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.570336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.570304    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:44.942810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:44.942780    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"]
Nov 03 21:26:44.944301 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:44.944273    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1edc2c9_4bee_4033_b865_58cad1e6613d.slice/crio-b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64 WatchSource:0}: Error finding container b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64: Status 404 returned error can't find the container with id b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64
Nov 03 21:26:45.671246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:45.671211    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7" event={"ID":"e1edc2c9-4bee-4033-b865-58cad1e6613d","Type":"ContainerStarted","Data":"b403e193fdbb3f7a236f681f41c702f2f8d79977a4a50ab728424ac8f529bdd1"}
Nov 03 21:26:45.671712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:45.671269    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7" event={"ID":"e1edc2c9-4bee-4033-b865-58cad1e6613d","Type":"ContainerStarted","Data":"b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64"}
Nov 03 21:26:45.780329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:45.780282    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7" podStartSLOduration=1.78026988 podStartE2EDuration="1.78026988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:45.779878793 +0000 UTC m=+2307.411192271" watchObservedRunningTime="2026-11-03 21:26:45.78026988 +0000 UTC m=+2307.411583356"
Nov 03 21:26:46.679065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:46.679021    5533 generic.go:356] "Generic (PLEG): container finished" podID="e1edc2c9-4bee-4033-b865-58cad1e6613d" containerID="b403e193fdbb3f7a236f681f41c702f2f8d79977a4a50ab728424ac8f529bdd1" exitCode=0
Nov 03 21:26:46.679439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:46.679102    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7" event={"ID":"e1edc2c9-4bee-4033-b865-58cad1e6613d","Type":"ContainerDied","Data":"b403e193fdbb3f7a236f681f41c702f2f8d79977a4a50ab728424ac8f529bdd1"}
Nov 03 21:26:47.840031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:47.840008    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:48.000303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.000224    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\") pod \"e1edc2c9-4bee-4033-b865-58cad1e6613d\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") "
Nov 03 21:26:48.000455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.000300    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\" (UniqueName: \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\") pod \"e1edc2c9-4bee-4033-b865-58cad1e6613d\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") "
Nov 03 21:26:48.000455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.000422    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\") pod \"e1edc2c9-4bee-4033-b865-58cad1e6613d\" (UID: \"e1edc2c9-4bee-4033-b865-58cad1e6613d\") "
Nov 03 21:26:48.000671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.000650    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca" pod "e1edc2c9-4bee-4033-b865-58cad1e6613d" (UID: "e1edc2c9-4bee-4033-b865-58cad1e6613d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:48.001073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.001045    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:48.002414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.002387    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92" pod "e1edc2c9-4bee-4033-b865-58cad1e6613d" (UID: "e1edc2c9-4bee-4033-b865-58cad1e6613d"). InnerVolumeSpecName "kube-api-access-ddg92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:48.002529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.002390    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca" pod "e1edc2c9-4bee-4033-b865-58cad1e6613d" (UID: "e1edc2c9-4bee-4033-b865-58cad1e6613d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:48.102269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.102241    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddg92\" (UniqueName: \"kubernetes.io/projected/e1edc2c9-4bee-4033-b865-58cad1e6613d-kube-api-access-ddg92\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:48.102269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.102268    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1edc2c9-4bee-4033-b865-58cad1e6613d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:48.694621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.694573    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7" event={"ID":"e1edc2c9-4bee-4033-b865-58cad1e6613d","Type":"ContainerDied","Data":"b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64"}
Nov 03 21:26:48.694833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.694669    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"
Nov 03 21:26:48.694833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:48.694723    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b618087da86aedda2a96bd4280c1daa47943b3c30f231ac1262bbb46df26cb64"
Nov 03 21:26:49.261268 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:26:49.261229    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:26:53.131133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:53.131091    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:26:55.022429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.022393    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"]
Nov 03 21:26:55.022935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.022919    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e1edc2c9-4bee-4033-b865-58cad1e6613d" containerName="revoke-keys"
Nov 03 21:26:55.022986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.022935    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1edc2c9-4bee-4033-b865-58cad1e6613d" containerName="revoke-keys"
Nov 03 21:26:55.023066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.023054    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1edc2c9-4bee-4033-b865-58cad1e6613d" containerName="revoke-keys"
Nov 03 21:26:55.026748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.026728    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.077696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.077672    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:26:55.077850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.077789    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:26:55.111275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.111248    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldpn\" (UniqueName: \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.111391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.111316    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.111453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.111418    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.127564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.127539    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"]
Nov 03 21:26:55.212338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.212313    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.212469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.212416    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldpn\" (UniqueName: \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.212536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.212512    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.213013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.212993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.214761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.214736    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.269601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.269575    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldpn\" (UniqueName: \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\") pod \"maas-api-revoke-sub-97d315fd-ktxrb\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.340142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.340092    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:55.707764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.707732    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"]
Nov 03 21:26:55.708620 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:26:55.708597    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd054fba9_a7be_429c_9c97_bcb4327f7494.slice/crio-48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973 WatchSource:0}: Error finding container 48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973: Status 404 returned error can't find the container with id 48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973
Nov 03 21:26:55.742333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:55.742305    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb" event={"ID":"d054fba9-a7be-429c-9c97-bcb4327f7494","Type":"ContainerStarted","Data":"48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973"}
Nov 03 21:26:56.749643 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:56.749610    5533 generic.go:356] "Generic (PLEG): container finished" podID="d054fba9-a7be-429c-9c97-bcb4327f7494" containerID="af820c1dbf32f20da95c7a73555f7ae3eb387f192f4c3dd81bcf4565b1204779" exitCode=0
Nov 03 21:26:56.750031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:56.749698    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb" event={"ID":"d054fba9-a7be-429c-9c97-bcb4327f7494","Type":"ContainerDied","Data":"af820c1dbf32f20da95c7a73555f7ae3eb387f192f4c3dd81bcf4565b1204779"}
Nov 03 21:26:57.909253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:57.909229    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:58.046734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.046613    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\") pod \"d054fba9-a7be-429c-9c97-bcb4327f7494\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") "
Nov 03 21:26:58.046734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.046691    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\") pod \"d054fba9-a7be-429c-9c97-bcb4327f7494\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") "
Nov 03 21:26:58.046984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.046754    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\" (UniqueName: \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\") pod \"d054fba9-a7be-429c-9c97-bcb4327f7494\" (UID: \"d054fba9-a7be-429c-9c97-bcb4327f7494\") "
Nov 03 21:26:58.047492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.047459    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca" pod "d054fba9-a7be-429c-9c97-bcb4327f7494" (UID: "d054fba9-a7be-429c-9c97-bcb4327f7494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:58.049250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.049221    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca" pod "d054fba9-a7be-429c-9c97-bcb4327f7494" (UID: "d054fba9-a7be-429c-9c97-bcb4327f7494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:58.049557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.049537    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn" pod "d054fba9-a7be-429c-9c97-bcb4327f7494" (UID: "d054fba9-a7be-429c-9c97-bcb4327f7494"). InnerVolumeSpecName "kube-api-access-nldpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:58.147897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.147870    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldpn\" (UniqueName: \"kubernetes.io/projected/d054fba9-a7be-429c-9c97-bcb4327f7494-kube-api-access-nldpn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:58.147897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.147895    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:58.148043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.147906    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d054fba9-a7be-429c-9c97-bcb4327f7494-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:26:58.765903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.765816    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"
Nov 03 21:26:58.765903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.765833    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb" event={"ID":"d054fba9-a7be-429c-9c97-bcb4327f7494","Type":"ContainerDied","Data":"48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973"}
Nov 03 21:26:58.765903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:26:58.765884    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a0628caad68d40ab3cbaee59338adc2efcca4ece3e41bbe69c7882b434e973"
Nov 03 21:27:11.224254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.224201    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"]
Nov 03 21:27:11.224878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.224858    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d054fba9-a7be-429c-9c97-bcb4327f7494" containerName="revoke-keys"
Nov 03 21:27:11.224943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.224880    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d054fba9-a7be-429c-9c97-bcb4327f7494" containerName="revoke-keys"
Nov 03 21:27:11.225041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.225030    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d054fba9-a7be-429c-9c97-bcb4327f7494" containerName="revoke-keys"
Nov 03 21:27:11.229859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.229840    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.282573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.282550    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"]
Nov 03 21:27:11.293741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.293709    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r57t\" (UniqueName: \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.293879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.293750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.293949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.293898    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.325279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.325254    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:11.325372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.325320    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:27:11.394690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.394665    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.394793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.394767    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r57t\" (UniqueName: \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.394870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.394804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.395353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.395333    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.397096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.397072    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.452248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.452224    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r57t\" (UniqueName: \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\") pod \"maas-api-revoke-sub-013dba96-dzhlh\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.544918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.544868    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:11.915295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:11.915257    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"]
Nov 03 21:27:11.916617 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:11.916587    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca7ae52_f4b7_4c32_9213_09eae40319b6.slice/crio-a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03 WatchSource:0}: Error finding container a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03: Status 404 returned error can't find the container with id a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03
Nov 03 21:27:12.864733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:12.864700    5533 generic.go:356] "Generic (PLEG): container finished" podID="1ca7ae52-f4b7-4c32-9213-09eae40319b6" containerID="e56fd9c6261b9fca13cab8833e388108cd18f5be6dfb4d24d1ef176eb2bfbdef" exitCode=0
Nov 03 21:27:12.865116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:12.864741    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh" event={"ID":"1ca7ae52-f4b7-4c32-9213-09eae40319b6","Type":"ContainerDied","Data":"e56fd9c6261b9fca13cab8833e388108cd18f5be6dfb4d24d1ef176eb2bfbdef"}
Nov 03 21:27:12.865116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:12.864797    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh" event={"ID":"1ca7ae52-f4b7-4c32-9213-09eae40319b6","Type":"ContainerStarted","Data":"a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03"}
Nov 03 21:27:14.020021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.019992    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:14.121420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.121386    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\") pod \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") "
Nov 03 21:27:14.121554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.121453    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\") pod \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") "
Nov 03 21:27:14.121620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.121560    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\" (UniqueName: \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\") pod \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\" (UID: \"1ca7ae52-f4b7-4c32-9213-09eae40319b6\") "
Nov 03 21:27:14.121877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.121826    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca" pod "1ca7ae52-f4b7-4c32-9213-09eae40319b6" (UID: "1ca7ae52-f4b7-4c32-9213-09eae40319b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:14.121973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.121955    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:14.123558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.123536    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t" pod "1ca7ae52-f4b7-4c32-9213-09eae40319b6" (UID: "1ca7ae52-f4b7-4c32-9213-09eae40319b6"). InnerVolumeSpecName "kube-api-access-8r57t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:14.123656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.123587    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca" pod "1ca7ae52-f4b7-4c32-9213-09eae40319b6" (UID: "1ca7ae52-f4b7-4c32-9213-09eae40319b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:14.222521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.222493    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7ae52-f4b7-4c32-9213-09eae40319b6-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:14.222521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.222519    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r57t\" (UniqueName: \"kubernetes.io/projected/1ca7ae52-f4b7-4c32-9213-09eae40319b6-kube-api-access-8r57t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:14.880425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.880341    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"
Nov 03 21:27:14.880577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.880352    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh" event={"ID":"1ca7ae52-f4b7-4c32-9213-09eae40319b6","Type":"ContainerDied","Data":"a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03"}
Nov 03 21:27:14.880577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:14.880475    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9a29902876bfe0512e56db54d883989440ad02ef442497c8a028eec56491f03"
Nov 03 21:27:16.146504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.146466    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"]
Nov 03 21:27:16.147245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.147222    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1ca7ae52-f4b7-4c32-9213-09eae40319b6" containerName="revoke-keys"
Nov 03 21:27:16.147245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.147245    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca7ae52-f4b7-4c32-9213-09eae40319b6" containerName="revoke-keys"
Nov 03 21:27:16.147434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.147420    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca7ae52-f4b7-4c32-9213-09eae40319b6" containerName="revoke-keys"
Nov 03 21:27:16.150994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.150971    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.247736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.247705    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:16.247923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.247778    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:27:16.251618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.251586    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"]
Nov 03 21:27:16.253428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.253399    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.253592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.253451    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.253592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.253511    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4q48\" (UniqueName: \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.355106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.355072    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.355262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.355124    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.355262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.355177    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4q48\" (UniqueName: \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.355883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.355856    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.357499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.357474    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.413161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.413102    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4q48\" (UniqueName: \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\") pod \"maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.476137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.476111    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:16.856041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.856004    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"]
Nov 03 21:27:16.856610 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:16.856583    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61090225_622e_41ed_9a48_fc441ea3879f.slice/crio-f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610 WatchSource:0}: Error finding container f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610: Status 404 returned error can't find the container with id f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610
Nov 03 21:27:16.896659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:16.896609    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw" event={"ID":"61090225-622e-41ed-9a48-fc441ea3879f","Type":"ContainerStarted","Data":"f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610"}
Nov 03 21:27:17.906576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:17.906539    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw" event={"ID":"61090225-622e-41ed-9a48-fc441ea3879f","Type":"ContainerStarted","Data":"dbdcd468d7a4041085799fce3e0ada84949d37f353f995a0141dc8643398dcac"}
Nov 03 21:27:18.018716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:18.018659    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw" podStartSLOduration=2.018627315 podStartE2EDuration="2.018627315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:18.015192373 +0000 UTC m=+2339.646505864" watchObservedRunningTime="2026-11-03 21:27:18.018627315 +0000 UTC m=+2339.649940793"
Nov 03 21:27:18.915073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:18.915030    5533 generic.go:356] "Generic (PLEG): container finished" podID="61090225-622e-41ed-9a48-fc441ea3879f" containerID="dbdcd468d7a4041085799fce3e0ada84949d37f353f995a0141dc8643398dcac" exitCode=0
Nov 03 21:27:18.915581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:18.915112    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw" event={"ID":"61090225-622e-41ed-9a48-fc441ea3879f","Type":"ContainerDied","Data":"dbdcd468d7a4041085799fce3e0ada84949d37f353f995a0141dc8643398dcac"}
Nov 03 21:27:19.261109 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:27:19.261074    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:27:19.819456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:19.819424    5533 scope.go:122] "RemoveContainer" containerID="2dddee8cdb02bd944e21dda953bf7d5a7c53f1f2a8b721496a4cd04dd74a44b4"
Nov 03 21:27:20.082384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.082311    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:20.202681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.202627    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\") pod \"61090225-622e-41ed-9a48-fc441ea3879f\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") "
Nov 03 21:27:20.202850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.202743    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\") pod \"61090225-622e-41ed-9a48-fc441ea3879f\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") "
Nov 03 21:27:20.202850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.202830    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\" (UniqueName: \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\") pod \"61090225-622e-41ed-9a48-fc441ea3879f\" (UID: \"61090225-622e-41ed-9a48-fc441ea3879f\") "
Nov 03 21:27:20.203947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.203919    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca" pod "61090225-622e-41ed-9a48-fc441ea3879f" (UID: "61090225-622e-41ed-9a48-fc441ea3879f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:20.205848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.205822    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca" pod "61090225-622e-41ed-9a48-fc441ea3879f" (UID: "61090225-622e-41ed-9a48-fc441ea3879f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:20.205955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.205843    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48" pod "61090225-622e-41ed-9a48-fc441ea3879f" (UID: "61090225-622e-41ed-9a48-fc441ea3879f"). InnerVolumeSpecName "kube-api-access-q4q48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:20.304203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.304166    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4q48\" (UniqueName: \"kubernetes.io/projected/61090225-622e-41ed-9a48-fc441ea3879f-kube-api-access-q4q48\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:20.304203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.304192    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:20.304203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.304202    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61090225-622e-41ed-9a48-fc441ea3879f-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:20.930942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.930911    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"
Nov 03 21:27:20.930942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.930924    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw" event={"ID":"61090225-622e-41ed-9a48-fc441ea3879f","Type":"ContainerDied","Data":"f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610"}
Nov 03 21:27:20.931157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:20.930970    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f42feec378390a0971fea145cc54651e51d3a681ce9d5de09616fed2da3ab610"
Nov 03 21:27:22.846067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.846035    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:27:22.846498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.846236    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy" containerID="cri-o://7ebd4383fbfd308a75752e1ee580b45fbc5794105acaff825ec526d01c7f947f" gracePeriod=30
Nov 03 21:27:22.894725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.894681    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:27:22.895481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.895452    5533 cpu_manager.go:414] "containerMap: removing container" podUID="61090225-622e-41ed-9a48-fc441ea3879f" containerName="revoke-keys"
Nov 03 21:27:22.895481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.895479    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61090225-622e-41ed-9a48-fc441ea3879f" containerName="revoke-keys"
Nov 03 21:27:22.895679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.895656    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="61090225-622e-41ed-9a48-fc441ea3879f" containerName="revoke-keys"
Nov 03 21:27:22.899181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.899163    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.936694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936665    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.936810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936708    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.936810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936746    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.936923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936847    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.936923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936878    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.937022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936963    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.937022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.936991    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt24\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.937135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.937091    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:22.937220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:22.937201    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.036727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.036686    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:27:23.044164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044123    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt24\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044216    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044293    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044341    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044387    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044423    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044502    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044531    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.044628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.044611    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.045342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.045315    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.045870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.045847    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.046370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.046351    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.047163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.047132    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.047432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.047409    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.094450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.094416    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.095314 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.095285    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.105709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.103760    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.105709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.104564    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt24\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.211961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.211932    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:23.681197 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:23.681157    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f138ead_4210_4a04_bdad_e00a622afd4a.slice/crio-8914449a6860f0b80dd9e4afd8692d51604477f68049ff47c70796c34889b38d WatchSource:0}: Error finding container 8914449a6860f0b80dd9e4afd8692d51604477f68049ff47c70796c34889b38d: Status 404 returned error can't find the container with id 8914449a6860f0b80dd9e4afd8692d51604477f68049ff47c70796c34889b38d
Nov 03 21:27:23.683915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.683878    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:23.683999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.683950    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:23.683999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.683989    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:23.685225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:23.685201    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:27:24.037787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:24.037707    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" event={"ID":"7f138ead-4210-4a04-bdad-e00a622afd4a","Type":"ContainerStarted","Data":"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"}
Nov 03 21:27:24.037787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:24.037766    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" event={"ID":"7f138ead-4210-4a04-bdad-e00a622afd4a","Type":"ContainerStarted","Data":"8914449a6860f0b80dd9e4afd8692d51604477f68049ff47c70796c34889b38d"}
Nov 03 21:27:24.213039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:24.213007    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:24.214734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:24.214707    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:24.214870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:24.214755    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:25.214265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:25.214216    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:25.214811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:25.214278    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:26.214668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:26.214292    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:26.214668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:26.214349    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:27.212661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.209938    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podStartSLOduration=5.209919852 podStartE2EDuration="5.209919852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:24.166713786 +0000 UTC m=+2345.798027263" watchObservedRunningTime="2026-11-03 21:27:27.209919852 +0000 UTC m=+2348.841233344"
Nov 03 21:27:27.213172 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.213145    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:27.214299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.214270    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:27.218703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.218669    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"]
Nov 03 21:27:27.227982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.227960    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.281657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.280611    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"]
Nov 03 21:27:27.320659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.319919    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:27.320659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.320628    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:27:27.340287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.340264    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.340518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.340502    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcgg\" (UniqueName: \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.340741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.340727    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.441812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.441773    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.441992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.441872    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.441992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.441910    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcgg\" (UniqueName: \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.442494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.442463    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.444431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.444413    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.511704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.509150    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcgg\" (UniqueName: \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\") pod \"maas-api-revoke-sub-a5401116-x6c8c\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.535294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.535260    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:27:27.539521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.539500    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.543292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.543272    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:27.607684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.600285    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3d317d\"/\"praxis-route-3d317d-kserve-self-signed-certs\""
Nov 03 21:27:27.633804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.632106    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3d317d\"/\"kube-root-ca.crt\""
Nov 03 21:27:27.633804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.632252    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3d317d\"/\"openshift-service-ca.crt\""
Nov 03 21:27:27.633804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.632837    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3d317d\"/\"default-dockercfg-45xzh\""
Nov 03 21:27:27.639761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.636443    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:27:27.678878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.674402    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.678878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.674466    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.678878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.674499    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.678878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.674538    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.678878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.674567    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.689863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.680934    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8g5w\" (UniqueName: \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782515    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8g5w\" (UniqueName: \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782687    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782730    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782760    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782792    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.782821    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.783391    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.783954    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.785681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.784192    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.808860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.808809    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.809264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.809199    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.841233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.841183    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8g5w\" (UniqueName: \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\") pod \"praxis-route-3d317d-kserve-85677854fc-t45hb\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") " pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:27.864740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:27.864495    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:28.103583 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.101406    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"]
Nov 03 21:27:28.117992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.117969    5533 generic.go:356] "Generic (PLEG): container finished" podID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerID="7ebd4383fbfd308a75752e1ee580b45fbc5794105acaff825ec526d01c7f947f" exitCode=0
Nov 03 21:27:28.118123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.118079    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerDied","Data":"7ebd4383fbfd308a75752e1ee580b45fbc5794105acaff825ec526d01c7f947f"}
Nov 03 21:27:28.118191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.118125    5533 scope.go:122] "RemoveContainer" containerID="3985ff90840d5c858cdc8f53dac3c028660a85aeebdf31dd420f79c8d6adc980"
Nov 03 21:27:28.212688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.212381    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:28.212688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.212434    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:28.346573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.346545    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:27:28.386785 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:28.386751    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99fb02a4_188f_4f84_a974_b62c5ed80653.slice/crio-6a134ad07caa0e9585f8a9cc860f74736319900b61958bb344a80865189a82bd WatchSource:0}: Error finding container 6a134ad07caa0e9585f8a9cc860f74736319900b61958bb344a80865189a82bd: Status 404 returned error can't find the container with id 6a134ad07caa0e9585f8a9cc860f74736319900b61958bb344a80865189a82bd
Nov 03 21:27:28.407377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.407353    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:27:28.512706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512677    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.512861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512719    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.512861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512771    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.512861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512806    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.512861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512824    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.513084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.512904    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.513084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513014    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.513202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513096    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.513202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513089    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:28.513202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513140    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\") pod \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\" (UID: \"2662e667-b1d9-4b9d-824f-0d8cec0305bc\") "
Nov 03 21:27:28.513202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513157    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:28.513423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513297    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:28.513485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513408    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:28.513540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513500    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:28.513734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513714    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.513825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513736    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.513825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513751    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.513825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513765    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.513825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.513779    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.515646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.515614    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:27:28.515806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.515784    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:27:28.516077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.516052    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "kube-api-access-gz5qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:28.622739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.571598    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token" pod "2662e667-b1d9-4b9d-824f-0d8cec0305bc" (UID: "2662e667-b1d9-4b9d-824f-0d8cec0305bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:28.622739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.615021    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5qr\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-kube-api-access-gz5qr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.622739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.615056    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.622739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.615068    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:28.622739 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:28.615399    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2662e667-b1d9-4b9d-824f-0d8cec0305bc-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:29.128449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.128326    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"
Nov 03 21:27:29.128449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.128370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc" event={"ID":"2662e667-b1d9-4b9d-824f-0d8cec0305bc","Type":"ContainerDied","Data":"07b66a0f78635bd4b2ecd8fa349cdfd87974c80135af6d31b717eeafdc8f2035"}
Nov 03 21:27:29.128449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.128432    5533 scope.go:122] "RemoveContainer" containerID="7ebd4383fbfd308a75752e1ee580b45fbc5794105acaff825ec526d01c7f947f"
Nov 03 21:27:29.130157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.130132    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" event={"ID":"99fb02a4-188f-4f84-a974-b62c5ed80653","Type":"ContainerStarted","Data":"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"}
Nov 03 21:27:29.130289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.130163    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" event={"ID":"99fb02a4-188f-4f84-a974-b62c5ed80653","Type":"ContainerStarted","Data":"6a134ad07caa0e9585f8a9cc860f74736319900b61958bb344a80865189a82bd"}
Nov 03 21:27:29.132113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.132089    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c" event={"ID":"d341475c-7919-4e67-b604-1b0178f46f66","Type":"ContainerStarted","Data":"c838c70cd52c78642264b65d2e91a09364e3b8843f16510c581a83a19ffba583"}
Nov 03 21:27:29.132304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.132124    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c" event={"ID":"d341475c-7919-4e67-b604-1b0178f46f66","Type":"ContainerStarted","Data":"2fb67df6e18dea5bb7abb1a540a0a7684fc5b7a92f013df1d012271857c1a4e2"}
Nov 03 21:27:29.212519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.212488    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:29.212715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.212540    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:29.297582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.297531    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" podStartSLOduration=2.297512702 podStartE2EDuration="2.297512702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:29.296919571 +0000 UTC m=+2350.928233049" watchObservedRunningTime="2026-11-03 21:27:29.297512702 +0000 UTC m=+2350.928826193"
Nov 03 21:27:29.406093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.406002    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:27:29.411252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.411217    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1dda2c-openshift-default-7fc7f4cdc7-252fc"]
Nov 03 21:27:29.518199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.518143    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c" podStartSLOduration=2.518126476 podStartE2EDuration="2.518126476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:29.515656403 +0000 UTC m=+2351.146969901" watchObservedRunningTime="2026-11-03 21:27:29.518126476 +0000 UTC m=+2351.149439953"
Nov 03 21:27:29.865269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:29.865219    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:30.145361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:30.145278    5533 generic.go:356] "Generic (PLEG): container finished" podID="d341475c-7919-4e67-b604-1b0178f46f66" containerID="c838c70cd52c78642264b65d2e91a09364e3b8843f16510c581a83a19ffba583" exitCode=0
Nov 03 21:27:30.145575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:30.145473    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c" event={"ID":"d341475c-7919-4e67-b604-1b0178f46f66","Type":"ContainerDied","Data":"c838c70cd52c78642264b65d2e91a09364e3b8843f16510c581a83a19ffba583"}
Nov 03 21:27:30.213053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:30.213016    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 21:27:30.213238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:30.213084    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 21:27:30.498886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:30.498843    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" path="/var/lib/kubelet/pods/2662e667-b1d9-4b9d-824f-0d8cec0305bc/volumes"
Nov 03 21:27:31.292024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.290784    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:31.292024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.291932    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:31.293908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.293844    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:27:31.331714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.331687    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:31.348566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.348539    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\" (UniqueName: \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\") pod \"d341475c-7919-4e67-b604-1b0178f46f66\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") "
Nov 03 21:27:31.348686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.348595    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\") pod \"d341475c-7919-4e67-b604-1b0178f46f66\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") "
Nov 03 21:27:31.348799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.348774    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\") pod \"d341475c-7919-4e67-b604-1b0178f46f66\" (UID: \"d341475c-7919-4e67-b604-1b0178f46f66\") "
Nov 03 21:27:31.349055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.349032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca" pod "d341475c-7919-4e67-b604-1b0178f46f66" (UID: "d341475c-7919-4e67-b604-1b0178f46f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:31.349619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.349275    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:31.350775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.350754    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca" pod "d341475c-7919-4e67-b604-1b0178f46f66" (UID: "d341475c-7919-4e67-b604-1b0178f46f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:31.350869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.350767    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg" pod "d341475c-7919-4e67-b604-1b0178f46f66" (UID: "d341475c-7919-4e67-b604-1b0178f46f66"). InnerVolumeSpecName "kube-api-access-vtcgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:31.450303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.450264    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcgg\" (UniqueName: \"kubernetes.io/projected/d341475c-7919-4e67-b604-1b0178f46f66-kube-api-access-vtcgg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:31.450303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:31.450305    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d341475c-7919-4e67-b604-1b0178f46f66-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:32.169027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.169000    5533 generic.go:356] "Generic (PLEG): container finished" podID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerID="d96cbf24862bf486c02f2d1c57c3f44b320f024c188cc15f29c04685ca70feaf" exitCode=0
Nov 03 21:27:32.169481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.169077    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerDied","Data":"d96cbf24862bf486c02f2d1c57c3f44b320f024c188cc15f29c04685ca70feaf"}
Nov 03 21:27:32.169543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.169508    5533 scope.go:122] "RemoveContainer" containerID="d96cbf24862bf486c02f2d1c57c3f44b320f024c188cc15f29c04685ca70feaf"
Nov 03 21:27:32.170960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.170851    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"
Nov 03 21:27:32.170960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.170883    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c" event={"ID":"d341475c-7919-4e67-b604-1b0178f46f66","Type":"ContainerDied","Data":"2fb67df6e18dea5bb7abb1a540a0a7684fc5b7a92f013df1d012271857c1a4e2"}
Nov 03 21:27:32.170960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.170934    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb67df6e18dea5bb7abb1a540a0a7684fc5b7a92f013df1d012271857c1a4e2"
Nov 03 21:27:32.171410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.171378    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:32.171500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.171470    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:32.171560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:32.171514    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:27:33.191486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:33.191445    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerStarted","Data":"38a927979f34a54e452ca6592ed8bee3febc4903a74fe9c11cc43abaf278fbd9"}
Nov 03 21:27:33.494249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:33.494152    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:27:33.495942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:33.495905    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 21:27:33.496097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:33.496025    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 21:27:34.301970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:34.301933    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:27:34.494503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:34.494473    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 21:27:34.494702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:34.494519    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 21:27:35.495281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:35.495244    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 21:27:35.495709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:35.495297    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 21:27:36.495086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:36.495049    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 21:27:36.495261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:36.495102    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 21:27:37.495126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.495085    5533 patch_prober.go:29] interesting pod/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 21:27:37.495517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.495142    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 21:27:37.646238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646202    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"]
Nov 03 21:27:37.646746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646725    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.646746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646745    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646761    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d341475c-7919-4e67-b604-1b0178f46f66" containerName="revoke-keys"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646766    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d341475c-7919-4e67-b604-1b0178f46f66" containerName="revoke-keys"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646780    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646786    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646910    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646919    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d341475c-7919-4e67-b604-1b0178f46f66" containerName="revoke-keys"
Nov 03 21:27:37.646957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.646931    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2662e667-b1d9-4b9d-824f-0d8cec0305bc" containerName="istio-proxy"
Nov 03 21:27:37.671656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.670868    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.703018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.702992    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"]
Nov 03 21:27:37.728510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.728485    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pmz\" (UniqueName: \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.728649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.728526    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.728701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.728651    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.745930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.745877    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:37.745930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.745922    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:27:37.830145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.830101    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pmz\" (UniqueName: \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.830145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.830158    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.830388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.830206    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.830857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.830837    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.832316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.832293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.875683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.874659    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:37.887189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.887164    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:37.887354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.887335    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pmz\" (UniqueName: \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\") pod \"maas-api-revoke-sub-a1937ac6-clmnd\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:37.987698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:37.987669    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:38.238108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:38.238077    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:27:38.387805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:38.387711    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"]
Nov 03 21:27:38.422964 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:38.422933    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea2860b_4206_43b2_a4cc_935b28da24ec.slice/crio-65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0 WatchSource:0}: Error finding container 65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0: Status 404 returned error can't find the container with id 65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0
Nov 03 21:27:38.509020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:38.508989    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:27:38.511388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:38.510951    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:27:39.236147 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:39.236104    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd" event={"ID":"fea2860b-4206-43b2-a4cc-935b28da24ec","Type":"ContainerStarted","Data":"21aae396a82634810f1d22d0ad37da0f4b142c4d8be2fa0d82beb7ecbf5571a4"}
Nov 03 21:27:39.236349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:39.236169    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd" event={"ID":"fea2860b-4206-43b2-a4cc-935b28da24ec","Type":"ContainerStarted","Data":"65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0"}
Nov 03 21:27:39.346579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:39.346533    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd" podStartSLOduration=2.346518883 podStartE2EDuration="2.346518883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:39.344307668 +0000 UTC m=+2360.975621145" watchObservedRunningTime="2026-11-03 21:27:39.346518883 +0000 UTC m=+2360.977832359"
Nov 03 21:27:40.244098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:40.244066    5533 generic.go:356] "Generic (PLEG): container finished" podID="fea2860b-4206-43b2-a4cc-935b28da24ec" containerID="21aae396a82634810f1d22d0ad37da0f4b142c4d8be2fa0d82beb7ecbf5571a4" exitCode=0
Nov 03 21:27:40.244478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:40.244133    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd" event={"ID":"fea2860b-4206-43b2-a4cc-935b28da24ec","Type":"ContainerDied","Data":"21aae396a82634810f1d22d0ad37da0f4b142c4d8be2fa0d82beb7ecbf5571a4"}
Nov 03 21:27:41.403564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.403537    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:41.469573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.469544    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\") pod \"fea2860b-4206-43b2-a4cc-935b28da24ec\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") "
Nov 03 21:27:41.469740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.469722    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\" (UniqueName: \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\") pod \"fea2860b-4206-43b2-a4cc-935b28da24ec\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") "
Nov 03 21:27:41.469798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.469776    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\") pod \"fea2860b-4206-43b2-a4cc-935b28da24ec\" (UID: \"fea2860b-4206-43b2-a4cc-935b28da24ec\") "
Nov 03 21:27:41.470135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.470105    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca" pod "fea2860b-4206-43b2-a4cc-935b28da24ec" (UID: "fea2860b-4206-43b2-a4cc-935b28da24ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:41.470267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.470247    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:41.471744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.471722    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca" pod "fea2860b-4206-43b2-a4cc-935b28da24ec" (UID: "fea2860b-4206-43b2-a4cc-935b28da24ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:41.471875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.471860    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz" pod "fea2860b-4206-43b2-a4cc-935b28da24ec" (UID: "fea2860b-4206-43b2-a4cc-935b28da24ec"). InnerVolumeSpecName "kube-api-access-v4pmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:41.571286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.571208    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pmz\" (UniqueName: \"kubernetes.io/projected/fea2860b-4206-43b2-a4cc-935b28da24ec-kube-api-access-v4pmz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:41.571446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:41.571313    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea2860b-4206-43b2-a4cc-935b28da24ec-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:42.259342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:42.259308    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"
Nov 03 21:27:42.259342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:42.259330    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd" event={"ID":"fea2860b-4206-43b2-a4cc-935b28da24ec","Type":"ContainerDied","Data":"65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0"}
Nov 03 21:27:42.259556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:42.259394    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e7047ca62b9f0d0a03ddcde33f4ad55378d0f5cc8e3571a662cbcaa3c939a0"
Nov 03 21:27:48.630112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.630080    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"]
Nov 03 21:27:48.630667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.630649    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fea2860b-4206-43b2-a4cc-935b28da24ec" containerName="revoke-keys"
Nov 03 21:27:48.630667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.630668    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea2860b-4206-43b2-a4cc-935b28da24ec" containerName="revoke-keys"
Nov 03 21:27:48.630802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.630793    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea2860b-4206-43b2-a4cc-935b28da24ec" containerName="revoke-keys"
Nov 03 21:27:48.634082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.634057    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.684622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.684593    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:27:48.690287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.690258    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"]
Nov 03 21:27:48.731336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.731319    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:27:48.742867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.742842    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2zq\" (UniqueName: \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.742935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.742911    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.743084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.743064    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.844104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.844080    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.844208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.844135    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2zq\" (UniqueName: \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.844208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.844181    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.844835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.844816    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.846372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.846355    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.902288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.902235    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2zq\" (UniqueName: \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\") pod \"maas-api-revoke-sub-534f4d17-bm2tk\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:48.947349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:48.947326    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:49.261120 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:27:49.261077    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:27:49.316192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:49.316163    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"]
Nov 03 21:27:49.316767 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:27:49.316742    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33aa2ed_c57c_447d_a89a_6e2208b6082a.slice/crio-e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f WatchSource:0}: Error finding container e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f: Status 404 returned error can't find the container with id e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f
Nov 03 21:27:50.328139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:50.328106    5533 generic.go:356] "Generic (PLEG): container finished" podID="a33aa2ed-c57c-447d-a89a-6e2208b6082a" containerID="6615dbd972dd042b92bfe10e573c9b9ab29205947162c1c0e0f9a8abb5038182" exitCode=0
Nov 03 21:27:50.328544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:50.328196    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk" event={"ID":"a33aa2ed-c57c-447d-a89a-6e2208b6082a","Type":"ContainerDied","Data":"6615dbd972dd042b92bfe10e573c9b9ab29205947162c1c0e0f9a8abb5038182"}
Nov 03 21:27:50.328544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:50.328253    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk" event={"ID":"a33aa2ed-c57c-447d-a89a-6e2208b6082a","Type":"ContainerStarted","Data":"e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f"}
Nov 03 21:27:51.423511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.423475    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:27:51.489880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.489849    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:51.577641 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.577618    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\" (UniqueName: \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\") pod \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") "
Nov 03 21:27:51.577783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.577680    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\") pod \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") "
Nov 03 21:27:51.577783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.577724    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\") pod \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\" (UID: \"a33aa2ed-c57c-447d-a89a-6e2208b6082a\") "
Nov 03 21:27:51.578159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.578136    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca" pod "a33aa2ed-c57c-447d-a89a-6e2208b6082a" (UID: "a33aa2ed-c57c-447d-a89a-6e2208b6082a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:51.579552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.579530    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq" pod "a33aa2ed-c57c-447d-a89a-6e2208b6082a" (UID: "a33aa2ed-c57c-447d-a89a-6e2208b6082a"). InnerVolumeSpecName "kube-api-access-fg2zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:51.579623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.579604    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca" pod "a33aa2ed-c57c-447d-a89a-6e2208b6082a" (UID: "a33aa2ed-c57c-447d-a89a-6e2208b6082a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:51.679759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.679726    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2zq\" (UniqueName: \"kubernetes.io/projected/a33aa2ed-c57c-447d-a89a-6e2208b6082a-kube-api-access-fg2zq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:51.679874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.679761    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:51.679874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:51.679778    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33aa2ed-c57c-447d-a89a-6e2208b6082a-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:27:52.345447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:52.345411    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"
Nov 03 21:27:52.345682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:52.345445    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk" event={"ID":"a33aa2ed-c57c-447d-a89a-6e2208b6082a","Type":"ContainerDied","Data":"e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f"}
Nov 03 21:27:52.345682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:27:52.345500    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20e503b846493cd29ccd7eec9cfdae4a4eb1b7579bd970801e38739e7e6aa4f"
Nov 03 21:28:03.088113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.088080    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"]
Nov 03 21:28:03.088617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.088603    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a33aa2ed-c57c-447d-a89a-6e2208b6082a" containerName="revoke-keys"
Nov 03 21:28:03.088697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.088618    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33aa2ed-c57c-447d-a89a-6e2208b6082a" containerName="revoke-keys"
Nov 03 21:28:03.088747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.088737    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33aa2ed-c57c-447d-a89a-6e2208b6082a" containerName="revoke-keys"
Nov 03 21:28:03.092672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.092650    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.123331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.123306    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.123482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.123349    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhjz\" (UniqueName: \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.123482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.123444    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.146013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.145988    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"]
Nov 03 21:28:03.188389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.188365    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:28:03.188575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.188552    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:28:03.225215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.225184    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.225378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.225362    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.225444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.225417    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhjz\" (UniqueName: \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.225865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.225842    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.227838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.227812    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.285050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.285026    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhjz\" (UniqueName: \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\") pod \"maas-api-revoke-sub-15a972e4-jb5tq\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.407431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.407372    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:03.778856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:03.778812    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"]
Nov 03 21:28:03.779054 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:03.779023    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c4d09c_08ba_4bb2_ac78_a3bed141bbd8.slice/crio-6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43 WatchSource:0}: Error finding container 6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43: Status 404 returned error can't find the container with id 6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43
Nov 03 21:28:04.442451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:04.442413    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq" event={"ID":"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8","Type":"ContainerStarted","Data":"8a91c0dc96c6dc1fba1f37f2085c65c030bd99ee2ffa86303a6fddb62c1b0582"}
Nov 03 21:28:04.442975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:04.442502    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq" event={"ID":"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8","Type":"ContainerStarted","Data":"6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43"}
Nov 03 21:28:04.555248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:04.555192    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq" podStartSLOduration=1.5551790319999999 podStartE2EDuration="1.555179032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:04.551765966 +0000 UTC m=+2386.183079443" watchObservedRunningTime="2026-11-03 21:28:04.555179032 +0000 UTC m=+2386.186492512"
Nov 03 21:28:05.452442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:05.452401    5533 generic.go:356] "Generic (PLEG): container finished" podID="06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" containerID="8a91c0dc96c6dc1fba1f37f2085c65c030bd99ee2ffa86303a6fddb62c1b0582" exitCode=0
Nov 03 21:28:05.452906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:05.452490    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq" event={"ID":"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8","Type":"ContainerDied","Data":"8a91c0dc96c6dc1fba1f37f2085c65c030bd99ee2ffa86303a6fddb62c1b0582"}
Nov 03 21:28:06.613815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.613789    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:06.662612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.662581    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\") pod \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") "
Nov 03 21:28:06.662815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.662746    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\" (UniqueName: \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\") pod \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") "
Nov 03 21:28:06.662815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.662809    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\") pod \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\" (UID: \"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8\") "
Nov 03 21:28:06.663620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.663586    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca" pod "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" (UID: "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:06.665270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.665244    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca" pod "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" (UID: "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:28:06.665375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.665264    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz" pod "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" (UID: "06c4d09c-08ba-4bb2-ac78-a3bed141bbd8"). InnerVolumeSpecName "kube-api-access-llhjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:06.764500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.764410    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:06.764500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.764448    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:06.764500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:06.764464    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhjz\" (UniqueName: \"kubernetes.io/projected/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8-kube-api-access-llhjz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:07.470464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:07.470428    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"
Nov 03 21:28:07.470658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:07.470459    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq" event={"ID":"06c4d09c-08ba-4bb2-ac78-a3bed141bbd8","Type":"ContainerDied","Data":"6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43"}
Nov 03 21:28:07.470658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:07.470518    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8d8fcd21e0818ad0128f93f58cff88c9607c536f2b2cf3f123c4afc585ff43"
Nov 03 21:28:10.248715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.248677    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"]
Nov 03 21:28:10.249196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.249182    5533 cpu_manager.go:414] "containerMap: removing container" podUID="06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" containerName="revoke-keys"
Nov 03 21:28:10.249260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.249198    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" containerName="revoke-keys"
Nov 03 21:28:10.249307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.249298    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" containerName="revoke-keys"
Nov 03 21:28:10.252918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.252895    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.303262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.303235    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.303379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.303342    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfwh\" (UniqueName: \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.303379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.303364    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.309110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.309084    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"]
Nov 03 21:28:10.364361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.364331    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:28:10.364945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.364616    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:28:10.404655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.404597    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.405549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.405522    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfwh\" (UniqueName: \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.406126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.406103    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.407109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.406922    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.410320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.410287    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.463824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.463786    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfwh\" (UniqueName: \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\") pod \"maas-api-revoke-sub-5e75a582-mbq2l\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.567388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.567318    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:10.938730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:10.938695    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"]
Nov 03 21:28:10.938921 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:10.938903    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1035f87b_6c1d_4203_b0be_1d8800c7eb3a.slice/crio-09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257 WatchSource:0}: Error finding container 09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257: Status 404 returned error can't find the container with id 09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257
Nov 03 21:28:11.418040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:11.417793    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"]
Nov 03 21:28:11.421155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:11.421131    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eae138-cb4tw"]
Nov 03 21:28:11.520749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:11.520666    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l" event={"ID":"1035f87b-6c1d-4203-b0be-1d8800c7eb3a","Type":"ContainerStarted","Data":"3b668384181eb84dd44d344def8f678e65d6db7c0f67ef3943301af7cfecdd05"}
Nov 03 21:28:11.520749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:11.520728    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l" event={"ID":"1035f87b-6c1d-4203-b0be-1d8800c7eb3a","Type":"ContainerStarted","Data":"09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257"}
Nov 03 21:28:11.631053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:11.631000    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l" podStartSLOduration=1.630986678 podStartE2EDuration="1.630986678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:11.6290055 +0000 UTC m=+2393.260318979" watchObservedRunningTime="2026-11-03 21:28:11.630986678 +0000 UTC m=+2393.262300186"
Nov 03 21:28:12.493332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:12.493300    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61090225-622e-41ed-9a48-fc441ea3879f" path="/var/lib/kubelet/pods/61090225-622e-41ed-9a48-fc441ea3879f/volumes"
Nov 03 21:28:12.529514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:12.529485    5533 generic.go:356] "Generic (PLEG): container finished" podID="1035f87b-6c1d-4203-b0be-1d8800c7eb3a" containerID="3b668384181eb84dd44d344def8f678e65d6db7c0f67ef3943301af7cfecdd05" exitCode=0
Nov 03 21:28:12.529712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:12.529538    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l" event={"ID":"1035f87b-6c1d-4203-b0be-1d8800c7eb3a","Type":"ContainerDied","Data":"3b668384181eb84dd44d344def8f678e65d6db7c0f67ef3943301af7cfecdd05"}
Nov 03 21:28:13.691840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.691811    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:13.745091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.745064    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\") pod \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") "
Nov 03 21:28:13.745223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.745161    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\") pod \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") "
Nov 03 21:28:13.745267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.745247    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\" (UniqueName: \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\") pod \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\" (UID: \"1035f87b-6c1d-4203-b0be-1d8800c7eb3a\") "
Nov 03 21:28:13.745442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.745421    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca" pod "1035f87b-6c1d-4203-b0be-1d8800c7eb3a" (UID: "1035f87b-6c1d-4203-b0be-1d8800c7eb3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:13.745692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.745662    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:13.747106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.747080    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca" pod "1035f87b-6c1d-4203-b0be-1d8800c7eb3a" (UID: "1035f87b-6c1d-4203-b0be-1d8800c7eb3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:28:13.747627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.747599    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh" pod "1035f87b-6c1d-4203-b0be-1d8800c7eb3a" (UID: "1035f87b-6c1d-4203-b0be-1d8800c7eb3a"). InnerVolumeSpecName "kube-api-access-rsfwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:13.846684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.846613    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:13.846684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:13.846653    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfwh\" (UniqueName: \"kubernetes.io/projected/1035f87b-6c1d-4203-b0be-1d8800c7eb3a-kube-api-access-rsfwh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:14.544669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:14.544623    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"
Nov 03 21:28:14.544811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:14.544661    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l" event={"ID":"1035f87b-6c1d-4203-b0be-1d8800c7eb3a","Type":"ContainerDied","Data":"09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257"}
Nov 03 21:28:14.544811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:14.544714    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a409e8c2314170a0bd7a33ea03f6b8ba519ae165152fded551f99de59e5257"
Nov 03 21:28:18.331293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.331235    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:28:18.439058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.439014    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:28:18.949518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.949490    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:28:18.962037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.962010    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:28:18.971891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.971872    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:28:18.984181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:18.984164    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:28:19.263339 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:28:19.263248    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:28:20.664305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.664270    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:28:20.664928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.664505    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" podUID="398a8e11-5956-4904-9118-203967ac073c" containerName="payload-pre-processing" containerID="cri-o://ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a" gracePeriod=30
Nov 03 21:28:20.918901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.918848    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:28:20.920698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.920669    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\" (UniqueName: \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\") pod \"398a8e11-5956-4904-9118-203967ac073c\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") "
Nov 03 21:28:20.920816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.920752    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\") pod \"398a8e11-5956-4904-9118-203967ac073c\" (UID: \"398a8e11-5956-4904-9118-203967ac073c\") "
Nov 03 21:28:20.921138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.921112    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume" pod "398a8e11-5956-4904-9118-203967ac073c" (UID: "398a8e11-5956-4904-9118-203967ac073c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:20.922700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:20.922678    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk" pod "398a8e11-5956-4904-9118-203967ac073c" (UID: "398a8e11-5956-4904-9118-203967ac073c"). InnerVolumeSpecName "kube-api-access-ww2gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:21.022519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.022492    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398a8e11-5956-4904-9118-203967ac073c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:21.022519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.022517    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2gk\" (UniqueName: \"kubernetes.io/projected/398a8e11-5956-4904-9118-203967ac073c-kube-api-access-ww2gk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:21.600738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.600708    5533 generic.go:356] "Generic (PLEG): container finished" podID="398a8e11-5956-4904-9118-203967ac073c" containerID="ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a" exitCode=0
Nov 03 21:28:21.600910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.600757    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"
Nov 03 21:28:21.600910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.600793    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" event={"ID":"398a8e11-5956-4904-9118-203967ac073c","Type":"ContainerDied","Data":"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"}
Nov 03 21:28:21.600910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.600845    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7" event={"ID":"398a8e11-5956-4904-9118-203967ac073c","Type":"ContainerDied","Data":"dc5ac6a4ff68fcb2fb1ee39a9ff1a33acd8e726abc0acf93c91142df5bf68198"}
Nov 03 21:28:21.600910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.600868    5533 scope.go:122] "RemoveContainer" containerID="ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"
Nov 03 21:28:21.610656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.610618    5533 scope.go:122] "RemoveContainer" containerID="ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"
Nov 03 21:28:21.610917 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:28:21.610898    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a\": container with ID starting with ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a not found: ID does not exist" containerID="ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"
Nov 03 21:28:21.610975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.610924    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a"} err="failed to get container status \"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a\": rpc error: code = NotFound desc = could not find container \"ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a\": container with ID starting with ca9d2ad8f70cc27f72d5e5dfdad0366fa1bd427381bf3f54bcf0a7b3dc5dc71a not found: ID does not exist"
Nov 03 21:28:21.757797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.757773    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:28:21.761495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:21.761474    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a77060-7cc4999844-4gqs7"]
Nov 03 21:28:22.493770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:22.493736    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398a8e11-5956-4904-9118-203967ac073c" path="/var/lib/kubelet/pods/398a8e11-5956-4904-9118-203967ac073c/volumes"
Nov 03 21:28:23.332039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.332001    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" podUID="1fda55b2-0134-4b76-87ca-522cacca8c84" containerName="payload-processing" containerID="cri-o://11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc" gracePeriod=25
Nov 03 21:28:23.439975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.439921    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" podUID="66a49d4c-5570-42ae-b76e-329d4e226d9a" containerName="payload-pre-processing" containerID="cri-o://90537bbef84f5968e4160aa0c40f1f9a25c6d52cccc17db04c6f703eee478ef9" gracePeriod=25
Nov 03 21:28:23.588694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.588624    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:28:23.621339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.621308    5533 generic.go:356] "Generic (PLEG): container finished" podID="1fda55b2-0134-4b76-87ca-522cacca8c84" containerID="11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc" exitCode=0
Nov 03 21:28:23.621452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.621436    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" event={"ID":"1fda55b2-0134-4b76-87ca-522cacca8c84","Type":"ContainerDied","Data":"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"}
Nov 03 21:28:23.621503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.621488    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m" event={"ID":"1fda55b2-0134-4b76-87ca-522cacca8c84","Type":"ContainerDied","Data":"27311286f75cce68b6dd68b7203fc8e7561836c74f1cec93a7edc805eae1072b"}
Nov 03 21:28:23.621541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.621521    5533 scope.go:122] "RemoveContainer" containerID="11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"
Nov 03 21:28:23.621653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.621618    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"
Nov 03 21:28:23.623775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.623749    5533 generic.go:356] "Generic (PLEG): container finished" podID="66a49d4c-5570-42ae-b76e-329d4e226d9a" containerID="90537bbef84f5968e4160aa0c40f1f9a25c6d52cccc17db04c6f703eee478ef9" exitCode=0
Nov 03 21:28:23.623893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.623795    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" event={"ID":"66a49d4c-5570-42ae-b76e-329d4e226d9a","Type":"ContainerDied","Data":"90537bbef84f5968e4160aa0c40f1f9a25c6d52cccc17db04c6f703eee478ef9"}
Nov 03 21:28:23.633670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.633625    5533 scope.go:122] "RemoveContainer" containerID="11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"
Nov 03 21:28:23.634034 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:28:23.634012    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc\": container with ID starting with 11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc not found: ID does not exist" containerID="11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"
Nov 03 21:28:23.634114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.634044    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc"} err="failed to get container status \"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc\": rpc error: code = NotFound desc = could not find container \"11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc\": container with ID starting with 11a0b79b84e4700f8d87467153fb71929b583120f73b647acbe88a5a708c34bc not found: ID does not exist"
Nov 03 21:28:23.659134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.659108    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\") pod \"1fda55b2-0134-4b76-87ca-522cacca8c84\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") "
Nov 03 21:28:23.659250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.659186    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\" (UniqueName: \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\") pod \"1fda55b2-0134-4b76-87ca-522cacca8c84\" (UID: \"1fda55b2-0134-4b76-87ca-522cacca8c84\") "
Nov 03 21:28:23.659534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.659508    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume" pod "1fda55b2-0134-4b76-87ca-522cacca8c84" (UID: "1fda55b2-0134-4b76-87ca-522cacca8c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:23.659714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.659697    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fda55b2-0134-4b76-87ca-522cacca8c84-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:23.661553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.661529    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd" pod "1fda55b2-0134-4b76-87ca-522cacca8c84" (UID: "1fda55b2-0134-4b76-87ca-522cacca8c84"). InnerVolumeSpecName "kube-api-access-86vxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:23.687215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.687197    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:28:23.760709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.760678    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\") pod \"66a49d4c-5570-42ae-b76e-329d4e226d9a\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") "
Nov 03 21:28:23.760832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.760757    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\") pod \"66a49d4c-5570-42ae-b76e-329d4e226d9a\" (UID: \"66a49d4c-5570-42ae-b76e-329d4e226d9a\") "
Nov 03 21:28:23.761116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.761092    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume" pod "66a49d4c-5570-42ae-b76e-329d4e226d9a" (UID: "66a49d4c-5570-42ae-b76e-329d4e226d9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:23.761246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.761231    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66a49d4c-5570-42ae-b76e-329d4e226d9a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:23.761291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.761252    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vxd\" (UniqueName: \"kubernetes.io/projected/1fda55b2-0134-4b76-87ca-522cacca8c84-kube-api-access-86vxd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:23.762705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.762687    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w" pod "66a49d4c-5570-42ae-b76e-329d4e226d9a" (UID: "66a49d4c-5570-42ae-b76e-329d4e226d9a"). InnerVolumeSpecName "kube-api-access-8nw8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:23.871801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:23.871734    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/66a49d4c-5570-42ae-b76e-329d4e226d9a-kube-api-access-8nw8w\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:24.037511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.037484    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:28:24.040665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.040628    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b5c926-76f74b9586-6tl4m"]
Nov 03 21:28:24.493918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.493889    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fda55b2-0134-4b76-87ca-522cacca8c84" path="/var/lib/kubelet/pods/1fda55b2-0134-4b76-87ca-522cacca8c84/volumes"
Nov 03 21:28:24.632367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.632293    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"
Nov 03 21:28:24.632367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.632308    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l" event={"ID":"66a49d4c-5570-42ae-b76e-329d4e226d9a","Type":"ContainerDied","Data":"f89aec861378bd0367b1492fc21ea895fe22ea286a08ab5bd4a45fde080aac77"}
Nov 03 21:28:24.632587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.632369    5533 scope.go:122] "RemoveContainer" containerID="90537bbef84f5968e4160aa0c40f1f9a25c6d52cccc17db04c6f703eee478ef9"
Nov 03 21:28:24.788756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.788726    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:28:24.835087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:24.835054    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b5c926-55df457495-d8l4l"]
Nov 03 21:28:25.973167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973132    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"]
Nov 03 21:28:25.973686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973669    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1fda55b2-0134-4b76-87ca-522cacca8c84" containerName="payload-processing"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973687    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fda55b2-0134-4b76-87ca-522cacca8c84" containerName="payload-processing"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973700    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1035f87b-6c1d-4203-b0be-1d8800c7eb3a" containerName="revoke-keys"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973706    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1035f87b-6c1d-4203-b0be-1d8800c7eb3a" containerName="revoke-keys"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973715    5533 cpu_manager.go:414] "containerMap: removing container" podUID="66a49d4c-5570-42ae-b76e-329d4e226d9a" containerName="payload-pre-processing"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973720    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a49d4c-5570-42ae-b76e-329d4e226d9a" containerName="payload-pre-processing"
Nov 03 21:28:25.973740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973740    5533 cpu_manager.go:414] "containerMap: removing container" podUID="398a8e11-5956-4904-9118-203967ac073c" containerName="payload-pre-processing"
Nov 03 21:28:25.973932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973746    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398a8e11-5956-4904-9118-203967ac073c" containerName="payload-pre-processing"
Nov 03 21:28:25.973932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973828    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fda55b2-0134-4b76-87ca-522cacca8c84" containerName="payload-processing"
Nov 03 21:28:25.973932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973837    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1035f87b-6c1d-4203-b0be-1d8800c7eb3a" containerName="revoke-keys"
Nov 03 21:28:25.973932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973844    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a49d4c-5570-42ae-b76e-329d4e226d9a" containerName="payload-pre-processing"
Nov 03 21:28:25.973932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.973851    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="398a8e11-5956-4904-9118-203967ac073c" containerName="payload-pre-processing"
Nov 03 21:28:25.978428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:25.978410    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.034090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.034059    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"]
Nov 03 21:28:26.075238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.075206    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:28:26.075238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.075207    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:28:26.095904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.095874    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtdc\" (UniqueName: \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.096076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.095915    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.096076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.095946    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.197254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.197227    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtdc\" (UniqueName: \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.197421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.197271    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.197421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.197338    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.198221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.198200    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.199734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.199713    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.255282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.255218    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtdc\" (UniqueName: \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\") pod \"maas-api-revoke-sub-b35a966c-pln5t\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.295494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.295466    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:26.494065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.494032    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a49d4c-5570-42ae-b76e-329d4e226d9a" path="/var/lib/kubelet/pods/66a49d4c-5570-42ae-b76e-329d4e226d9a/volumes"
Nov 03 21:28:26.669382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:26.669352    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"]
Nov 03 21:28:26.670424 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:26.670392    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6257d3b6_086d_45fa_af3b_bc9f45cd547e.slice/crio-53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932 WatchSource:0}: Error finding container 53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932: Status 404 returned error can't find the container with id 53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932
Nov 03 21:28:27.659283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:27.659252    5533 generic.go:356] "Generic (PLEG): container finished" podID="6257d3b6-086d-45fa-af3b-bc9f45cd547e" containerID="9d6bd64807b5307c7caf3929737430ee065e78d7788a21ab2b925d00730c0c7e" exitCode=0
Nov 03 21:28:27.659722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:27.659343    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t" event={"ID":"6257d3b6-086d-45fa-af3b-bc9f45cd547e","Type":"ContainerDied","Data":"9d6bd64807b5307c7caf3929737430ee065e78d7788a21ab2b925d00730c0c7e"}
Nov 03 21:28:27.659722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:27.659444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t" event={"ID":"6257d3b6-086d-45fa-af3b-bc9f45cd547e","Type":"ContainerStarted","Data":"53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932"}
Nov 03 21:28:28.885464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.885432    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:28.927755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.927693    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\") pod \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") "
Nov 03 21:28:28.927886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.927770    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\") pod \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") "
Nov 03 21:28:28.927886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.927804    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\" (UniqueName: \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\") pod \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\" (UID: \"6257d3b6-086d-45fa-af3b-bc9f45cd547e\") "
Nov 03 21:28:28.928119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.928095    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca" pod "6257d3b6-086d-45fa-af3b-bc9f45cd547e" (UID: "6257d3b6-086d-45fa-af3b-bc9f45cd547e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:28.928334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.928312    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:28.929877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.929853    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca" pod "6257d3b6-086d-45fa-af3b-bc9f45cd547e" (UID: "6257d3b6-086d-45fa-af3b-bc9f45cd547e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:28:28.929877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:28.929869    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc" pod "6257d3b6-086d-45fa-af3b-bc9f45cd547e" (UID: "6257d3b6-086d-45fa-af3b-bc9f45cd547e"). InnerVolumeSpecName "kube-api-access-vvtdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:29.029478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:29.029446    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6257d3b6-086d-45fa-af3b-bc9f45cd547e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:29.029478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:29.029472    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtdc\" (UniqueName: \"kubernetes.io/projected/6257d3b6-086d-45fa-af3b-bc9f45cd547e-kube-api-access-vvtdc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:29.678004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:29.677970    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"
Nov 03 21:28:29.678004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:29.677999    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t" event={"ID":"6257d3b6-086d-45fa-af3b-bc9f45cd547e","Type":"ContainerDied","Data":"53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932"}
Nov 03 21:28:29.678208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:29.678049    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cad83adc983e9b1b0d56d35a7802dfdb902396b7164daccfac9b8d72706932"
Nov 03 21:28:40.920405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:40.920354    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:28:41.809473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.809432    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:28:41.810225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.810203    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6257d3b6-086d-45fa-af3b-bc9f45cd547e" containerName="revoke-keys"
Nov 03 21:28:41.810358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.810228    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6257d3b6-086d-45fa-af3b-bc9f45cd547e" containerName="revoke-keys"
Nov 03 21:28:41.810429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.810416    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6257d3b6-086d-45fa-af3b-bc9f45cd547e" containerName="revoke-keys"
Nov 03 21:28:41.816231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.816212    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.866185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.866159    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-791c32-openshift-default-dockercfg-zfm58\""
Nov 03 21:28:41.926741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.926708    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:28:41.982624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.982590    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.982974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.982948    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pd7m\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.983164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.983142    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.983348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.983330    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.983557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.983536    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.983773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.983754    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.983997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.983979    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.984217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.984199    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:41.984381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:41.984364    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.105566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.105471    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.106369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.106344    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pd7m\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.106896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.106873    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.107093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.107060    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.107263 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.107245    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.107433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.107414    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.107603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.107587    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.110885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.110864    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.111010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.110994    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.111195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.108185    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.111308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.108797    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.111404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.109001    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.130652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.106209    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.130652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.107946    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.140800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.140205    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.156679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.155006    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.193243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.175917    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pd7m\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.193243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.177297    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.433006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.432980    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:42.887687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.887161    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:28:42.902476 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:42.902444    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc661d9b0_04e9_4d64_953b_8e1be4b50929.slice/crio-b1a6b50a9bfcdf7fe9d6a18694ba2db74caf2357f420cdabb3fb49c426c520f8 WatchSource:0}: Error finding container b1a6b50a9bfcdf7fe9d6a18694ba2db74caf2357f420cdabb3fb49c426c520f8: Status 404 returned error can't find the container with id b1a6b50a9bfcdf7fe9d6a18694ba2db74caf2357f420cdabb3fb49c426c520f8
Nov 03 21:28:42.904901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.904871    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:28:42.904986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.904936    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:28:42.905028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:42.904987    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:28:43.793359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:43.793317    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" event={"ID":"c661d9b0-04e9-4d64-953b-8e1be4b50929","Type":"ContainerStarted","Data":"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"}
Nov 03 21:28:43.793843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:43.793402    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" event={"ID":"c661d9b0-04e9-4d64-953b-8e1be4b50929","Type":"ContainerStarted","Data":"b1a6b50a9bfcdf7fe9d6a18694ba2db74caf2357f420cdabb3fb49c426c520f8"}
Nov 03 21:28:43.910005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:43.909950    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podStartSLOduration=2.909935054 podStartE2EDuration="2.909935054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:43.906767032 +0000 UTC m=+2425.538080521" watchObservedRunningTime="2026-11-03 21:28:43.909935054 +0000 UTC m=+2425.541248531"
Nov 03 21:28:44.434156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:44.434127    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:44.435458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:44.435435    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:44.435584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:44.435503    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:45.367660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.366988    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"]
Nov 03 21:28:45.381333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.381308    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.421192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.421156    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"]
Nov 03 21:28:45.433783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.433756    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:45.433916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.433810    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:45.470656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.464527    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:28:45.470656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.464689    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:28:45.550230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.550204    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.550376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.550347    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.550471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.550455    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6czm\" (UniqueName: \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.664669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.664560    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.664839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.664760    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.664916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.664884    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6czm\" (UniqueName: \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.669656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.668521    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.697113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.697084    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.722817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.722797    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6czm\" (UniqueName: \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\") pod \"maas-api-revoke-sub-afd31cc2-6cmld\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:45.995483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:45.995452    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:46.400818 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:46.400783    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc01fee_cfab_4640_8c4d_617e6df1458b.slice/crio-add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568 WatchSource:0}: Error finding container add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568: Status 404 returned error can't find the container with id add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568
Nov 03 21:28:46.401766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.401737    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"]
Nov 03 21:28:46.433537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.433513    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:46.433628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.433557    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:46.820052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.820010    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld" event={"ID":"5dc01fee-cfab-4640-8c4d-617e6df1458b","Type":"ContainerStarted","Data":"76bb45a7ab8ed6b32742cb81c7e5e34d54ebdcb2dfebee7a604c68c36ff7af4e"}
Nov 03 21:28:46.820194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.820111    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld" event={"ID":"5dc01fee-cfab-4640-8c4d-617e6df1458b","Type":"ContainerStarted","Data":"add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568"}
Nov 03 21:28:46.937807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:46.937752    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld" podStartSLOduration=1.937737654 podStartE2EDuration="1.937737654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:46.935082422 +0000 UTC m=+2428.566395898" watchObservedRunningTime="2026-11-03 21:28:46.937737654 +0000 UTC m=+2428.569051132"
Nov 03 21:28:47.434138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:47.434105    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:47.434592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:47.434167    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:47.832145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:47.831814    5533 generic.go:356] "Generic (PLEG): container finished" podID="5dc01fee-cfab-4640-8c4d-617e6df1458b" containerID="76bb45a7ab8ed6b32742cb81c7e5e34d54ebdcb2dfebee7a604c68c36ff7af4e" exitCode=0
Nov 03 21:28:47.832145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:47.831891    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld" event={"ID":"5dc01fee-cfab-4640-8c4d-617e6df1458b","Type":"ContainerDied","Data":"76bb45a7ab8ed6b32742cb81c7e5e34d54ebdcb2dfebee7a604c68c36ff7af4e"}
Nov 03 21:28:48.434261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:48.434230    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:48.434685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:48.434280    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:49.000893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.000860    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:49.022007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.021985    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\" (UniqueName: \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\") pod \"5dc01fee-cfab-4640-8c4d-617e6df1458b\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") "
Nov 03 21:28:49.022126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.022074    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\") pod \"5dc01fee-cfab-4640-8c4d-617e6df1458b\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") "
Nov 03 21:28:49.022126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.022103    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\") pod \"5dc01fee-cfab-4640-8c4d-617e6df1458b\" (UID: \"5dc01fee-cfab-4640-8c4d-617e6df1458b\") "
Nov 03 21:28:49.022436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.022403    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca" pod "5dc01fee-cfab-4640-8c4d-617e6df1458b" (UID: "5dc01fee-cfab-4640-8c4d-617e6df1458b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:49.022513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.022490    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:49.024114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.024086    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm" pod "5dc01fee-cfab-4640-8c4d-617e6df1458b" (UID: "5dc01fee-cfab-4640-8c4d-617e6df1458b"). InnerVolumeSpecName "kube-api-access-l6czm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:49.024318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.024295    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca" pod "5dc01fee-cfab-4640-8c4d-617e6df1458b" (UID: "5dc01fee-cfab-4640-8c4d-617e6df1458b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:28:49.124104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.124021    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc01fee-cfab-4640-8c4d-617e6df1458b-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:49.124104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.124053    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6czm\" (UniqueName: \"kubernetes.io/projected/5dc01fee-cfab-4640-8c4d-617e6df1458b-kube-api-access-l6czm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:49.263431 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:28:49.263389    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:28:49.433947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.433920    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 21:28:49.434111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.433978    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 21:28:49.667291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.667223    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"]
Nov 03 21:28:49.668698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.668671    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5dc01fee-cfab-4640-8c4d-617e6df1458b" containerName="revoke-keys"
Nov 03 21:28:49.668857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.668843    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc01fee-cfab-4640-8c4d-617e6df1458b" containerName="revoke-keys"
Nov 03 21:28:49.669105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.669091    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc01fee-cfab-4640-8c4d-617e6df1458b" containerName="revoke-keys"
Nov 03 21:28:49.673383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.673364    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.728109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.728080    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"]
Nov 03 21:28:49.730570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.730540    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.730712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.730626    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx2b\" (UniqueName: \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.730872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.730855    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.832024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.831990    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.832220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.832076    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.832220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.832149    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx2b\" (UniqueName: \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.832970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.832943    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.834376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.834347    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.847134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.847116    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"
Nov 03 21:28:49.847249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.847155    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld" event={"ID":"5dc01fee-cfab-4640-8c4d-617e6df1458b","Type":"ContainerDied","Data":"add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568"}
Nov 03 21:28:49.847249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.847212    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add7016028acda16601739036f1837823ec535f24f2bfb79eb8a5034626f4568"
Nov 03 21:28:49.890818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.890778    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx2b\" (UniqueName: \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\") pod \"maas-api-revoke-sub-6590b997-lvwvc\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:49.996813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:49.996728    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:50.386861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.386828    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"]
Nov 03 21:28:50.388721 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:28:50.388692    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d123c24_13bf_480f_8121_2c618acc230e.slice/crio-9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6 WatchSource:0}: Error finding container 9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6: Status 404 returned error can't find the container with id 9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6
Nov 03 21:28:50.437933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.437906    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:50.438322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.438300    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:50.439125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.439106    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:28:50.857439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.857392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc" event={"ID":"9d123c24-13bf-480f-8121-2c618acc230e","Type":"ContainerStarted","Data":"9a86522a3a3b9869c87e0d403c5a2cc5e4805950a96a314451a2575e9b2749c1"}
Nov 03 21:28:50.857909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.857456    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc" event={"ID":"9d123c24-13bf-480f-8121-2c618acc230e","Type":"ContainerStarted","Data":"9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6"}
Nov 03 21:28:50.970794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:50.970744    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc" podStartSLOduration=1.970729821 podStartE2EDuration="1.970729821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:50.96759622 +0000 UTC m=+2432.598909701" watchObservedRunningTime="2026-11-03 21:28:50.970729821 +0000 UTC m=+2432.602043297"
Nov 03 21:28:51.866455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:51.866419    5533 generic.go:356] "Generic (PLEG): container finished" podID="9d123c24-13bf-480f-8121-2c618acc230e" containerID="9a86522a3a3b9869c87e0d403c5a2cc5e4805950a96a314451a2575e9b2749c1" exitCode=0
Nov 03 21:28:51.866957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:51.866471    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc" event={"ID":"9d123c24-13bf-480f-8121-2c618acc230e","Type":"ContainerDied","Data":"9a86522a3a3b9869c87e0d403c5a2cc5e4805950a96a314451a2575e9b2749c1"}
Nov 03 21:28:53.014343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.014315    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:53.064929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.064900    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\") pod \"9d123c24-13bf-480f-8121-2c618acc230e\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") "
Nov 03 21:28:53.065076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.065039    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\") pod \"9d123c24-13bf-480f-8121-2c618acc230e\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") "
Nov 03 21:28:53.065122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.065113    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\" (UniqueName: \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\") pod \"9d123c24-13bf-480f-8121-2c618acc230e\" (UID: \"9d123c24-13bf-480f-8121-2c618acc230e\") "
Nov 03 21:28:53.065499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.065471    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca" pod "9d123c24-13bf-480f-8121-2c618acc230e" (UID: "9d123c24-13bf-480f-8121-2c618acc230e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:53.066832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.066808    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca" pod "9d123c24-13bf-480f-8121-2c618acc230e" (UID: "9d123c24-13bf-480f-8121-2c618acc230e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:28:53.067192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.067170    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b" pod "9d123c24-13bf-480f-8121-2c618acc230e" (UID: "9d123c24-13bf-480f-8121-2c618acc230e"). InnerVolumeSpecName "kube-api-access-vhx2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:53.167183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.167103    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:53.167183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.167138    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhx2b\" (UniqueName: \"kubernetes.io/projected/9d123c24-13bf-480f-8121-2c618acc230e-kube-api-access-vhx2b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:53.167183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.167148    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d123c24-13bf-480f-8121-2c618acc230e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:53.776378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.776345    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:28:53.881654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.881610    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"
Nov 03 21:28:53.881801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.881669    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc" event={"ID":"9d123c24-13bf-480f-8121-2c618acc230e","Type":"ContainerDied","Data":"9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6"}
Nov 03 21:28:53.881801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.881710    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d995154a3aac35708864d7d2df38c61182ea65aed941acffec91c477843c9d6"
Nov 03 21:28:53.886159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:53.886135    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:28:58.777408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:58.777360    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" podUID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" containerName="payload-processing" containerID="cri-o://5533f9f7f676b120951a59619df1eae93c3fd785cd78465a80c565693da4d4cb" gracePeriod=25
Nov 03 21:28:58.886666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:58.886614    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" podUID="de213a03-d758-4007-a2d3-5777d052377b" containerName="payload-pre-processing" containerID="cri-o://cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7" gracePeriod=25
Nov 03 21:28:58.919345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:58.919318    5533 generic.go:356] "Generic (PLEG): container finished" podID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" containerID="5533f9f7f676b120951a59619df1eae93c3fd785cd78465a80c565693da4d4cb" exitCode=0
Nov 03 21:28:58.919451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:58.919395    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" event={"ID":"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2","Type":"ContainerDied","Data":"5533f9f7f676b120951a59619df1eae93c3fd785cd78465a80c565693da4d4cb"}
Nov 03 21:28:59.053485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.053458    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:28:59.133681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.133655    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\" (UniqueName: \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\") pod \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") "
Nov 03 21:28:59.133862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.133847    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\") pod \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\" (UID: \"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2\") "
Nov 03 21:28:59.134303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.134281    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume" pod "f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" (UID: "f06ff874-c3ef-439d-91d4-ab9dfdddb3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:59.135665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.135623    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2" pod "f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" (UID: "f06ff874-c3ef-439d-91d4-ab9dfdddb3b2"). InnerVolumeSpecName "kube-api-access-zhqr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:59.144320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.144300    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:28:59.234626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.234598    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\" (UniqueName: \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\") pod \"de213a03-d758-4007-a2d3-5777d052377b\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") "
Nov 03 21:28:59.234777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.234761    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\") pod \"de213a03-d758-4007-a2d3-5777d052377b\" (UID: \"de213a03-d758-4007-a2d3-5777d052377b\") "
Nov 03 21:28:59.235142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.235119    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume" pod "de213a03-d758-4007-a2d3-5777d052377b" (UID: "de213a03-d758-4007-a2d3-5777d052377b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:28:59.235274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.235184    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:59.235274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.235227    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhqr2\" (UniqueName: \"kubernetes.io/projected/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2-kube-api-access-zhqr2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:59.236845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.236825    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw" pod "de213a03-d758-4007-a2d3-5777d052377b" (UID: "de213a03-d758-4007-a2d3-5777d052377b"). InnerVolumeSpecName "kube-api-access-wbfhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:28:59.336099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.336035    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbfhw\" (UniqueName: \"kubernetes.io/projected/de213a03-d758-4007-a2d3-5777d052377b-kube-api-access-wbfhw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:59.336099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.336062    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de213a03-d758-4007-a2d3-5777d052377b-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:28:59.932397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.932358    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"
Nov 03 21:28:59.932896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.932391    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9" event={"ID":"f06ff874-c3ef-439d-91d4-ab9dfdddb3b2","Type":"ContainerDied","Data":"2a29edb9f91eb1d645083a165c7c2f3f2347428ba0e449a52e80a353ac802832"}
Nov 03 21:28:59.932896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.932462    5533 scope.go:122] "RemoveContainer" containerID="5533f9f7f676b120951a59619df1eae93c3fd785cd78465a80c565693da4d4cb"
Nov 03 21:28:59.933933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.933913    5533 generic.go:356] "Generic (PLEG): container finished" podID="de213a03-d758-4007-a2d3-5777d052377b" containerID="cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7" exitCode=0
Nov 03 21:28:59.934010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.933955    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"
Nov 03 21:28:59.934077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.934052    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" event={"ID":"de213a03-d758-4007-a2d3-5777d052377b","Type":"ContainerDied","Data":"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"}
Nov 03 21:28:59.934140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.934110    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf" event={"ID":"de213a03-d758-4007-a2d3-5777d052377b","Type":"ContainerDied","Data":"84a7308eb977a2919ce5e2ab42e76e47f8a05694dc4e25bef8fe321d6e36028e"}
Nov 03 21:28:59.943162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.943142    5533 scope.go:122] "RemoveContainer" containerID="cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"
Nov 03 21:28:59.953153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.953134    5533 scope.go:122] "RemoveContainer" containerID="cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"
Nov 03 21:28:59.953437 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:28:59.953417    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7\": container with ID starting with cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7 not found: ID does not exist" containerID="cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"
Nov 03 21:28:59.953505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:28:59.953450    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7"} err="failed to get container status \"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7\": rpc error: code = NotFound desc = could not find container \"cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7\": container with ID starting with cd2c625588455ef6ea3ba22baabf3d14717ccf4681f7f9dfc96011ce6c21a0b7 not found: ID does not exist"
Nov 03 21:29:00.088667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.088564    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:29:00.096114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.096082    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2285b9-799997959-v7jgf"]
Nov 03 21:29:00.197404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.197371    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:29:00.202844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.202817    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2285b9-7fc7b456db-nkzm9"]
Nov 03 21:29:00.494560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.494527    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de213a03-d758-4007-a2d3-5777d052377b" path="/var/lib/kubelet/pods/de213a03-d758-4007-a2d3-5777d052377b/volumes"
Nov 03 21:29:00.495136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.495115    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" path="/var/lib/kubelet/pods/f06ff874-c3ef-439d-91d4-ab9dfdddb3b2/volumes"
Nov 03 21:29:00.645658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.645612    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"]
Nov 03 21:29:00.646129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646114    5533 cpu_manager.go:414] "containerMap: removing container" podUID="de213a03-d758-4007-a2d3-5777d052377b" containerName="payload-pre-processing"
Nov 03 21:29:00.646177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646130    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de213a03-d758-4007-a2d3-5777d052377b" containerName="payload-pre-processing"
Nov 03 21:29:00.646177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646155    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" containerName="payload-processing"
Nov 03 21:29:00.646177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646160    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" containerName="payload-processing"
Nov 03 21:29:00.646177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646171    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9d123c24-13bf-480f-8121-2c618acc230e" containerName="revoke-keys"
Nov 03 21:29:00.646177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646176    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d123c24-13bf-480f-8121-2c618acc230e" containerName="revoke-keys"
Nov 03 21:29:00.646459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646253    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="de213a03-d758-4007-a2d3-5777d052377b" containerName="payload-pre-processing"
Nov 03 21:29:00.646459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646264    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06ff874-c3ef-439d-91d4-ab9dfdddb3b2" containerName="payload-processing"
Nov 03 21:29:00.646459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.646272    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d123c24-13bf-480f-8121-2c618acc230e" containerName="revoke-keys"
Nov 03 21:29:00.651281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.651262    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.700363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.700339    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:00.700547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.700525    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:29:00.703857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.703827    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"]
Nov 03 21:29:00.752748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.752695    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.752834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.752761    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.752875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.752857    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbs47\" (UniqueName: \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.854400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.854377    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.854505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.854446    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.854505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.854474    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbs47\" (UniqueName: \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.855153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.855130    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.856686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.856657    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.912485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.912455    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbs47\" (UniqueName: \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\") pod \"maas-api-revoke-sub-2c4a63e9-vqzvk\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:00.966543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:00.966513    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:01.334980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:01.334953    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"]
Nov 03 21:29:01.335900 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:01.335864    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bc6efd_fa21_4f18_af58_fbfff88f4acd.slice/crio-377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152 WatchSource:0}: Error finding container 377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152: Status 404 returned error can't find the container with id 377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152
Nov 03 21:29:01.952810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:01.952765    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk" event={"ID":"33bc6efd-fa21-4f18-af58-fbfff88f4acd","Type":"ContainerStarted","Data":"5c4711d32b7049d5536513361a4510d8444a0a84256c13055213c6108cc09383"}
Nov 03 21:29:01.952994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:01.952829    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk" event={"ID":"33bc6efd-fa21-4f18-af58-fbfff88f4acd","Type":"ContainerStarted","Data":"377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152"}
Nov 03 21:29:02.064326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:02.064271    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk" podStartSLOduration=2.064257242 podStartE2EDuration="2.064257242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:02.061117693 +0000 UTC m=+2443.692431172" watchObservedRunningTime="2026-11-03 21:29:02.064257242 +0000 UTC m=+2443.695570718"
Nov 03 21:29:02.961767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:02.961733    5533 generic.go:356] "Generic (PLEG): container finished" podID="33bc6efd-fa21-4f18-af58-fbfff88f4acd" containerID="5c4711d32b7049d5536513361a4510d8444a0a84256c13055213c6108cc09383" exitCode=0
Nov 03 21:29:02.961942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:02.961822    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk" event={"ID":"33bc6efd-fa21-4f18-af58-fbfff88f4acd","Type":"ContainerDied","Data":"5c4711d32b7049d5536513361a4510d8444a0a84256c13055213c6108cc09383"}
Nov 03 21:29:04.113358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.113335    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:04.193551 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.193524    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\" (UniqueName: \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\") pod \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") "
Nov 03 21:29:04.193733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.193663    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\") pod \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") "
Nov 03 21:29:04.193808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.193736    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\") pod \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\" (UID: \"33bc6efd-fa21-4f18-af58-fbfff88f4acd\") "
Nov 03 21:29:04.194132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.194107    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca" pod "33bc6efd-fa21-4f18-af58-fbfff88f4acd" (UID: "33bc6efd-fa21-4f18-af58-fbfff88f4acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:04.194272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.194254    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:04.195661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.195613    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47" pod "33bc6efd-fa21-4f18-af58-fbfff88f4acd" (UID: "33bc6efd-fa21-4f18-af58-fbfff88f4acd"). InnerVolumeSpecName "kube-api-access-pbs47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:04.195774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.195692    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca" pod "33bc6efd-fa21-4f18-af58-fbfff88f4acd" (UID: "33bc6efd-fa21-4f18-af58-fbfff88f4acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:04.295065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.294991    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc6efd-fa21-4f18-af58-fbfff88f4acd-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:04.295065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.295021    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbs47\" (UniqueName: \"kubernetes.io/projected/33bc6efd-fa21-4f18-af58-fbfff88f4acd-kube-api-access-pbs47\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:04.976995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.976963    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"
Nov 03 21:29:04.977202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.977002    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk" event={"ID":"33bc6efd-fa21-4f18-af58-fbfff88f4acd","Type":"ContainerDied","Data":"377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152"}
Nov 03 21:29:04.977202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:04.977051    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377cb97c660d3abe60146900815df8efeb2579c039559d30501626b62056e152"
Nov 03 21:29:05.079652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.079607    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"]
Nov 03 21:29:05.080138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.080123    5533 cpu_manager.go:414] "containerMap: removing container" podUID="33bc6efd-fa21-4f18-af58-fbfff88f4acd" containerName="revoke-keys"
Nov 03 21:29:05.080184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.080139    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bc6efd-fa21-4f18-af58-fbfff88f4acd" containerName="revoke-keys"
Nov 03 21:29:05.080255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.080246    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bc6efd-fa21-4f18-af58-fbfff88f4acd" containerName="revoke-keys"
Nov 03 21:29:05.083842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.083827    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.133386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.133361    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:05.133900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.133878    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:29:05.150996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.150969    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"]
Nov 03 21:29:05.206169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.206141    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.206306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.206176    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.206306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.206293    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k792x\" (UniqueName: \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.307265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.307207    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.307265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.307248    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k792x\" (UniqueName: \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.307437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.307354    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.307981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.307963    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.309475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.309452    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.365217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.365190    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k792x\" (UniqueName: \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\") pod \"maas-api-revoke-sub-e9f9212a-sthtv\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.397906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.397873    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:05.773098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.773069    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"]
Nov 03 21:29:05.774569 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:05.774524    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9fd7e3_3c93_4d99_908e_84f25f3fa2d7.slice/crio-60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24 WatchSource:0}: Error finding container 60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24: Status 404 returned error can't find the container with id 60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24
Nov 03 21:29:05.988026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:05.987993    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv" event={"ID":"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7","Type":"ContainerStarted","Data":"60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24"}
Nov 03 21:29:06.997824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:06.997784    5533 generic.go:356] "Generic (PLEG): container finished" podID="8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" containerID="b29ef64a79b9ad68cc72558e3a0cd697c3e53758d5f2a7da70655ed3282818ed" exitCode=0
Nov 03 21:29:06.998239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:06.997869    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv" event={"ID":"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7","Type":"ContainerDied","Data":"b29ef64a79b9ad68cc72558e3a0cd697c3e53758d5f2a7da70655ed3282818ed"}
Nov 03 21:29:08.151814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.151784    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:08.236852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.236825    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\") pod \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") "
Nov 03 21:29:08.236991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.236861    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\") pod \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") "
Nov 03 21:29:08.237078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.237059    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\" (UniqueName: \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\") pod \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\" (UID: \"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7\") "
Nov 03 21:29:08.237229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.237201    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca" pod "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" (UID: "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:08.237453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.237438    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:08.238904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.238881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca" pod "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" (UID: "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:08.239080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.239061    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x" pod "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" (UID: "8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7"). InnerVolumeSpecName "kube-api-access-k792x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:08.338435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.338366    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k792x\" (UniqueName: \"kubernetes.io/projected/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-kube-api-access-k792x\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:08.338435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.338396    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:08.751236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.751203    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:29:08.751815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.751795    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" containerName="revoke-keys"
Nov 03 21:29:08.751815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.751813    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" containerName="revoke-keys"
Nov 03 21:29:08.751964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.751913    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" containerName="revoke-keys"
Nov 03 21:29:08.755463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.755441    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844204    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844344    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844379    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844430    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844592    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844680    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844713    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844741    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.844825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.844782    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.869544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.869513    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:29:08.945618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945565    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.945822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945629    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.945822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945684    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.945822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945721    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.945822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945748    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945840    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945936    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.945966    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.946022    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.946567    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.946885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.946862    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.947159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.947136    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.947405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.947384    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.948455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.948049    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.948455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.948382    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:08.960673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:08.960456    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:09.003442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.003375    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:09.005156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.005137    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:09.012444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.012427    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"
Nov 03 21:29:09.012544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.012470    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv" event={"ID":"8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7","Type":"ContainerDied","Data":"60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24"}
Nov 03 21:29:09.012600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.012560    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b1fa5026fff57d07021ebe5864d900b4f4029ee9d24a92eedbe7d27882de24"
Nov 03 21:29:09.072492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.072471    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:09.501701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.501667    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:29:09.502820 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:09.502787    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8788126a_0f5a_48d5_bdbb_92ddf4074cb3.slice/crio-9614c066a644080d824b68a5482bf21509e12813fdc922b675b52971b26ff575 WatchSource:0}: Error finding container 9614c066a644080d824b68a5482bf21509e12813fdc922b675b52971b26ff575: Status 404 returned error can't find the container with id 9614c066a644080d824b68a5482bf21509e12813fdc922b675b52971b26ff575
Nov 03 21:29:09.505843 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.505811    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:09.505947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.505875    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:09.505947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:09.505910    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:10.020323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.020282    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" event={"ID":"8788126a-0f5a-48d5-bdbb-92ddf4074cb3","Type":"ContainerStarted","Data":"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"}
Nov 03 21:29:10.020490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.020404    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" event={"ID":"8788126a-0f5a-48d5-bdbb-92ddf4074cb3","Type":"ContainerStarted","Data":"9614c066a644080d824b68a5482bf21509e12813fdc922b675b52971b26ff575"}
Nov 03 21:29:10.072744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.072713    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:10.074220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.074193    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:10.074359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.074239    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:10.137473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:10.137430    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podStartSLOduration=2.137416964 podStartE2EDuration="2.137416964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:10.133366587 +0000 UTC m=+2451.764680064" watchObservedRunningTime="2026-11-03 21:29:10.137416964 +0000 UTC m=+2451.768730440"
Nov 03 21:29:11.073660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:11.073047    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:11.073660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:11.073107    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:12.074023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:12.073985    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:12.074593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:12.074559    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:13.086660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:13.083015    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:13.086660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:13.083104    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:14.076659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:14.072850    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:14.076659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:14.072918    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:15.075975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:15.075932    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:15.076438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:15.075989    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:16.073738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:16.073697    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 21:29:16.073896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:16.073768    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 21:29:18.115796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:18.115752    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:29:18.116262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:18.115818    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:29:18.383498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:18.383401    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:18.384669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:18.384629    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:18.385410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:18.385387    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:29:19.262850 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:29:19.262818    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:29:22.011874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.011835    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"]
Nov 03 21:29:22.016415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.016392    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.070241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.070214    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fz6\" (UniqueName: \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.070358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.070339    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.070405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.070373    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.112232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.112209    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:29:22.112324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.112283    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:22.118219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.118197    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"]
Nov 03 21:29:22.171755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.171734    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.171841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.171768    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.171881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.171846    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fz6\" (UniqueName: \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.172410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.172387    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.174117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.174098    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.229649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.229621    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fz6\" (UniqueName: \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\") pod \"maas-api-revoke-sub-24d5a6a6-psc55\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.329738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.329686    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:22.549905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.549872    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"]
Nov 03 21:29:22.554945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.554924    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.576513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.576493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.576611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.576520    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.576833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.576808    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96xrg\" (UniqueName: \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.608621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.608567    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"]
Nov 03 21:29:22.677922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.677893    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96xrg\" (UniqueName: \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.678029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.677993    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.678072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.678025    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.678727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.678710    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.680315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.680298    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.714046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.714022    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"]
Nov 03 21:29:22.714178 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:22.714148    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb1aeee_de4b_425d_8f98_3c977246c615.slice/crio-e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71 WatchSource:0}: Error finding container e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71: Status 404 returned error can't find the container with id e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71
Nov 03 21:29:22.735185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.735163    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96xrg\" (UniqueName: \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\") pod \"maas-api-revoke-sub-0e54e645-g84ll\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:22.868827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:22.868778    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:23.155490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:23.155402    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55" event={"ID":"5fb1aeee-de4b-425d-8f98-3c977246c615","Type":"ContainerStarted","Data":"3153b65e5caf597f295c6b89809ed60699fa78575972e78250c96f197c6d36dd"}
Nov 03 21:29:23.155490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:23.155459    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55" event={"ID":"5fb1aeee-de4b-425d-8f98-3c977246c615","Type":"ContainerStarted","Data":"e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71"}
Nov 03 21:29:23.236411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:23.236383    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"]
Nov 03 21:29:23.237480 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:23.237444    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8582affc_2bd8_4d4f_9ef7_fc280faf6aba.slice/crio-f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd WatchSource:0}: Error finding container f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd: Status 404 returned error can't find the container with id f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd
Nov 03 21:29:23.266456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:23.266416    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55" podStartSLOduration=2.266402991 podStartE2EDuration="2.266402991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:23.264105614 +0000 UTC m=+2464.895419090" watchObservedRunningTime="2026-11-03 21:29:23.266402991 +0000 UTC m=+2464.897716487"
Nov 03 21:29:24.166810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:24.166782    5533 generic.go:356] "Generic (PLEG): container finished" podID="5fb1aeee-de4b-425d-8f98-3c977246c615" containerID="3153b65e5caf597f295c6b89809ed60699fa78575972e78250c96f197c6d36dd" exitCode=0
Nov 03 21:29:24.167196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:24.166872    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55" event={"ID":"5fb1aeee-de4b-425d-8f98-3c977246c615","Type":"ContainerDied","Data":"3153b65e5caf597f295c6b89809ed60699fa78575972e78250c96f197c6d36dd"}
Nov 03 21:29:24.168778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:24.168754    5533 generic.go:356] "Generic (PLEG): container finished" podID="8582affc-2bd8-4d4f-9ef7-fc280faf6aba" containerID="d1a380f5388745a7381a98b023d04fb365200d1f9e9cde0afb48452c836ab578" exitCode=0
Nov 03 21:29:24.168991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:24.168822    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll" event={"ID":"8582affc-2bd8-4d4f-9ef7-fc280faf6aba","Type":"ContainerDied","Data":"d1a380f5388745a7381a98b023d04fb365200d1f9e9cde0afb48452c836ab578"}
Nov 03 21:29:24.168991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:24.168855    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll" event={"ID":"8582affc-2bd8-4d4f-9ef7-fc280faf6aba","Type":"ContainerStarted","Data":"f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd"}
Nov 03 21:29:25.347929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.347673    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:25.379126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.379098    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:25.406326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406302    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\") pod \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") "
Nov 03 21:29:25.406472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406340    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\") pod \"5fb1aeee-de4b-425d-8f98-3c977246c615\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") "
Nov 03 21:29:25.406548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406478    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\") pod \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") "
Nov 03 21:29:25.406548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406524    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\") pod \"5fb1aeee-de4b-425d-8f98-3c977246c615\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") "
Nov 03 21:29:25.406684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406607    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\" (UniqueName: \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\") pod \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\" (UID: \"8582affc-2bd8-4d4f-9ef7-fc280faf6aba\") "
Nov 03 21:29:25.406751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406730    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\" (UniqueName: \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\") pod \"5fb1aeee-de4b-425d-8f98-3c977246c615\" (UID: \"5fb1aeee-de4b-425d-8f98-3c977246c615\") "
Nov 03 21:29:25.406930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406893    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca" pod "8582affc-2bd8-4d4f-9ef7-fc280faf6aba" (UID: "8582affc-2bd8-4d4f-9ef7-fc280faf6aba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:25.406930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.406909    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca" pod "5fb1aeee-de4b-425d-8f98-3c977246c615" (UID: "5fb1aeee-de4b-425d-8f98-3c977246c615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:25.407261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.407239    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:25.407347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.407266    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:25.408601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.408579    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca" pod "8582affc-2bd8-4d4f-9ef7-fc280faf6aba" (UID: "8582affc-2bd8-4d4f-9ef7-fc280faf6aba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:25.408818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.408797    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca" pod "5fb1aeee-de4b-425d-8f98-3c977246c615" (UID: "5fb1aeee-de4b-425d-8f98-3c977246c615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:25.409220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.409196    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6" pod "5fb1aeee-de4b-425d-8f98-3c977246c615" (UID: "5fb1aeee-de4b-425d-8f98-3c977246c615"). InnerVolumeSpecName "kube-api-access-r8fz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:25.409308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.409218    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg" pod "8582affc-2bd8-4d4f-9ef7-fc280faf6aba" (UID: "8582affc-2bd8-4d4f-9ef7-fc280faf6aba"). InnerVolumeSpecName "kube-api-access-96xrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:25.508565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.508493    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:25.508565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.508528    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fb1aeee-de4b-425d-8f98-3c977246c615-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:25.508565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.508544    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96xrg\" (UniqueName: \"kubernetes.io/projected/8582affc-2bd8-4d4f-9ef7-fc280faf6aba-kube-api-access-96xrg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:25.508565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:25.508560    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8fz6\" (UniqueName: \"kubernetes.io/projected/5fb1aeee-de4b-425d-8f98-3c977246c615-kube-api-access-r8fz6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:26.186038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.186008    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"
Nov 03 21:29:26.186038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.186023    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55" event={"ID":"5fb1aeee-de4b-425d-8f98-3c977246c615","Type":"ContainerDied","Data":"e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71"}
Nov 03 21:29:26.186262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.186076    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48399d63a747175a72d6d4a1e0df3aaaec4cba60b52c2104cd1c4129b604d71"
Nov 03 21:29:26.187624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.187607    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"
Nov 03 21:29:26.187750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.187660    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll" event={"ID":"8582affc-2bd8-4d4f-9ef7-fc280faf6aba","Type":"ContainerDied","Data":"f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd"}
Nov 03 21:29:26.187750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:26.187697    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f943c207804a31daaf5feeda629d4c8c0e8a9903b641d4673d13d89d0bcafd"
Nov 03 21:29:33.281355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.281316    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"]
Nov 03 21:29:33.282104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282081    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5fb1aeee-de4b-425d-8f98-3c977246c615" containerName="revoke-keys"
Nov 03 21:29:33.282104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282101    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb1aeee-de4b-425d-8f98-3c977246c615" containerName="revoke-keys"
Nov 03 21:29:33.282297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282138    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8582affc-2bd8-4d4f-9ef7-fc280faf6aba" containerName="revoke-keys"
Nov 03 21:29:33.282297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282146    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8582affc-2bd8-4d4f-9ef7-fc280faf6aba" containerName="revoke-keys"
Nov 03 21:29:33.282410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282317    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb1aeee-de4b-425d-8f98-3c977246c615" containerName="revoke-keys"
Nov 03 21:29:33.282410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.282335    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8582affc-2bd8-4d4f-9ef7-fc280faf6aba" containerName="revoke-keys"
Nov 03 21:29:33.286749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.286687    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.331811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.331786    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"]
Nov 03 21:29:33.373229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.373206    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:29:33.373378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.373245    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:33.387184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.387151    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.387328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.387311    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.387389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.387335    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjnl\" (UniqueName: \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.488599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.488563    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.488789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.488609    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjnl\" (UniqueName: \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.488789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.488723    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.489305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.489277    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.491279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.491253    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.546103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.546044    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjnl\" (UniqueName: \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\") pod \"maas-api-revoke-sub-008dad2c-8zwgx\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:33.609332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:33.608909    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:34.005827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:34.005730    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"]
Nov 03 21:29:34.289938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:34.289906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx" event={"ID":"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e","Type":"ContainerStarted","Data":"16efa40c489b7cada68c978b09142be750620ae3d365bac23c64f404d9639b4b"}
Nov 03 21:29:34.290344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:34.289965    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx" event={"ID":"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e","Type":"ContainerStarted","Data":"2e919037a64c0824bb737171678e6374487d181670f43e3b3a2bb8eb62d7e6e3"}
Nov 03 21:29:34.400560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:34.400514    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx" podStartSLOduration=1.400499484 podStartE2EDuration="1.400499484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:34.398544108 +0000 UTC m=+2476.029857586" watchObservedRunningTime="2026-11-03 21:29:34.400499484 +0000 UTC m=+2476.031812961"
Nov 03 21:29:35.298135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:35.298105    5533 generic.go:356] "Generic (PLEG): container finished" podID="5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" containerID="16efa40c489b7cada68c978b09142be750620ae3d365bac23c64f404d9639b4b" exitCode=0
Nov 03 21:29:35.298566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:35.298181    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx" event={"ID":"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e","Type":"ContainerDied","Data":"16efa40c489b7cada68c978b09142be750620ae3d365bac23c64f404d9639b4b"}
Nov 03 21:29:36.453660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.453622    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:36.539378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.539346    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\") pod \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") "
Nov 03 21:29:36.539562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.539460    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\") pod \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") "
Nov 03 21:29:36.539562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.539543    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\" (UniqueName: \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\") pod \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\" (UID: \"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e\") "
Nov 03 21:29:36.539803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.539775    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca" pod "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" (UID: "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:36.539954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.539935    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:36.573234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.573192    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca" pod "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" (UID: "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:36.574011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.573987    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl" pod "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" (UID: "5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e"). InnerVolumeSpecName "kube-api-access-fnjnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:36.641127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.641093    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:36.641127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:36.641122    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjnl\" (UniqueName: \"kubernetes.io/projected/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e-kube-api-access-fnjnl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:37.313895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:37.313862    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"
Nov 03 21:29:37.314079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:37.313893    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx" event={"ID":"5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e","Type":"ContainerDied","Data":"2e919037a64c0824bb737171678e6374487d181670f43e3b3a2bb8eb62d7e6e3"}
Nov 03 21:29:37.314079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:37.313941    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e919037a64c0824bb737171678e6374487d181670f43e3b3a2bb8eb62d7e6e3"
Nov 03 21:29:37.603290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:37.603198    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:29:39.338836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:39.338789    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:29:39.354724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:39.354689    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy" containerID="cri-o://38a927979f34a54e452ca6592ed8bee3febc4903a74fe9c11cc43abaf278fbd9" gracePeriod=30
Nov 03 21:29:44.487751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.487722    5533 generic.go:356] "Generic (PLEG): container finished" podID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerID="38a927979f34a54e452ca6592ed8bee3febc4903a74fe9c11cc43abaf278fbd9" exitCode=0
Nov 03 21:29:44.498478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.498445    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerDied","Data":"38a927979f34a54e452ca6592ed8bee3febc4903a74fe9c11cc43abaf278fbd9"}
Nov 03 21:29:44.498604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.498525    5533 scope.go:122] "RemoveContainer" containerID="d96cbf24862bf486c02f2d1c57c3f44b320f024c188cc15f29c04685ca70feaf"
Nov 03 21:29:44.573723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.573698    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:29:44.669124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669046    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669086    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669115    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669154    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669215    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669252    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669279    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669342    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:29:44.669680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669384    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669420    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\") pod \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\" (UID: \"09bc06c5-3d49-47d9-ba9d-9154b10e0c98\") "
Nov 03 21:29:44.669680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669546    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:29:44.669680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669625    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:29:44.669680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669558    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:29:44.669944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669916    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.669944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669932    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.670052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669933    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:44.670052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669942    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.670052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.669985    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.671624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.671599    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:29:44.671760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.671664    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:29:44.671859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.671844    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:44.672056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.672033    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk" pod "09bc06c5-3d49-47d9-ba9d-9154b10e0c98" (UID: "09bc06c5-3d49-47d9-ba9d-9154b10e0c98"). InnerVolumeSpecName "kube-api-access-r8nrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:44.770749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.770716    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.770749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.770744    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nrk\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-kube-api-access-r8nrk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.770749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.770755    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.771057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.770765    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:44.771057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:44.770774    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bc06c5-3d49-47d9-ba9d-9154b10e0c98-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:45.319712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.319679    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 21:29:45.320278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320254    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" containerName="revoke-keys"
Nov 03 21:29:45.320278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320275    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" containerName="revoke-keys"
Nov 03 21:29:45.320529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320312    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.320529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320319    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.320529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320490    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.320529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320508    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.320529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.320525    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" containerName="revoke-keys"
Nov 03 21:29:45.324104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.324084    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.374410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.374378    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c70fca-dockercfg-c8h7n\""
Nov 03 21:29:45.374930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.374908    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c70fca\""
Nov 03 21:29:45.377144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.377117    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.377252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.377214    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rq6\" (UniqueName: \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.380256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.380227    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 21:29:45.481661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.479048    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.481661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.479162    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rq6\" (UniqueName: \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.481661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.480450    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.496685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.484686    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 21:29:45.496685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.485728    5533 cpu_manager.go:414] "containerMap: removing container" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.496685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.485751    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" containerName="istio-proxy"
Nov 03 21:29:45.496685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.490377    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.507560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.504494    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2" event={"ID":"09bc06c5-3d49-47d9-ba9d-9154b10e0c98","Type":"ContainerDied","Data":"4da106620fd155cb021812a9642c711c9f5cda94c51b4d6e662de608a0a5cd28"}
Nov 03 21:29:45.507560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.504555    5533 scope.go:122] "RemoveContainer" containerID="38a927979f34a54e452ca6592ed8bee3febc4903a74fe9c11cc43abaf278fbd9"
Nov 03 21:29:45.507560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.504726    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"
Nov 03 21:29:45.529967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.525073    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 21:29:45.537705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.537677    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rq6\" (UniqueName: \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\") pod \"payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.580753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.580693    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.580994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.580971    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcjl\" (UniqueName: \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.638007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.637983    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:45.681855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.681828    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcjl\" (UniqueName: \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.681990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.681874    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.682589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.682565    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.714780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.705979    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:29:45.714780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.710899    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2285b9-openshift-default-575b44f585-zsmw2"]
Nov 03 21:29:45.741823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.741057    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcjl\" (UniqueName: \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\") pod \"payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:45.831337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:45.831270    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:46.013546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.013516    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 21:29:46.014779 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:46.014749    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c811d2_bf9e_4076_b729_23324a24d3c8.slice/crio-8059303dc97ca7058bb6c637c2f154a79bf8cb6df87cccc372ceff987bf9ae5e WatchSource:0}: Error finding container 8059303dc97ca7058bb6c637c2f154a79bf8cb6df87cccc372ceff987bf9ae5e: Status 404 returned error can't find the container with id 8059303dc97ca7058bb6c637c2f154a79bf8cb6df87cccc372ceff987bf9ae5e
Nov 03 21:29:46.204276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.204251    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 21:29:46.205148 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:46.205118    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7471ab21_70ed_4fb1_91c4_84caadfeb05a.slice/crio-79d3afe2960147b7ad688fb63a4108fa1b0fa3249fb276477997c22faa7cacc7 WatchSource:0}: Error finding container 79d3afe2960147b7ad688fb63a4108fa1b0fa3249fb276477997c22faa7cacc7: Status 404 returned error can't find the container with id 79d3afe2960147b7ad688fb63a4108fa1b0fa3249fb276477997c22faa7cacc7
Nov 03 21:29:46.496812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.496729    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bc06c5-3d49-47d9-ba9d-9154b10e0c98" path="/var/lib/kubelet/pods/09bc06c5-3d49-47d9-ba9d-9154b10e0c98/volumes"
Nov 03 21:29:46.513725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.513681    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" event={"ID":"7471ab21-70ed-4fb1-91c4-84caadfeb05a","Type":"ContainerStarted","Data":"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"}
Nov 03 21:29:46.513911 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.513747    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" event={"ID":"7471ab21-70ed-4fb1-91c4-84caadfeb05a","Type":"ContainerStarted","Data":"79d3afe2960147b7ad688fb63a4108fa1b0fa3249fb276477997c22faa7cacc7"}
Nov 03 21:29:46.513997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.513912    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:46.515388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.515361    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" event={"ID":"f5c811d2-bf9e-4076-b729-23324a24d3c8","Type":"ContainerStarted","Data":"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"}
Nov 03 21:29:46.515496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.515416    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:46.515496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.515442    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" event={"ID":"f5c811d2-bf9e-4076-b729-23324a24d3c8","Type":"ContainerStarted","Data":"8059303dc97ca7058bb6c637c2f154a79bf8cb6df87cccc372ceff987bf9ae5e"}
Nov 03 21:29:46.625835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.625787    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" podStartSLOduration=1.625773509 podStartE2EDuration="1.625773509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:46.625393932 +0000 UTC m=+2488.256707410" watchObservedRunningTime="2026-11-03 21:29:46.625773509 +0000 UTC m=+2488.257086987"
Nov 03 21:29:46.738349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:46.738301    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" podStartSLOduration=1.738287141 podStartE2EDuration="1.738287141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:46.734250788 +0000 UTC m=+2488.365564266" watchObservedRunningTime="2026-11-03 21:29:46.738287141 +0000 UTC m=+2488.369600665"
Nov 03 21:29:47.953630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:47.953597    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:29:49.263400 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:29:49.263363    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:29:50.554313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.554281    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"]
Nov 03 21:29:50.558517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.558500    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.612090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.612064    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"]
Nov 03 21:29:50.635956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.635925    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.636153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.636132    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhtp\" (UniqueName: \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.636213 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.636196    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.652821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.652794    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:29:50.654132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.654113    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:29:50.737269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.737247    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhtp\" (UniqueName: \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.737401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.737289    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.737401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.737317    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.737978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.737956    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.739619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.739598    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.794625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.794606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhtp\" (UniqueName: \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\") pod \"maas-api-revoke-sub-12c08fe4-bvv6j\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:50.872070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:50.872000    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:51.249408 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:51.249368    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b774c1_1ed3_4a04_9861_244ba96d259c.slice/crio-26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737 WatchSource:0}: Error finding container 26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737: Status 404 returned error can't find the container with id 26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737
Nov 03 21:29:51.251601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:51.251571    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"]
Nov 03 21:29:51.564949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:51.564926    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j" event={"ID":"d8b774c1-1ed3-4a04-9861-244ba96d259c","Type":"ContainerStarted","Data":"26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737"}
Nov 03 21:29:52.573906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:52.573869    5533 generic.go:356] "Generic (PLEG): container finished" podID="d8b774c1-1ed3-4a04-9861-244ba96d259c" containerID="40fd00de6a396be35caf8a01a53cc2cf6640c1a557972fc535b6ee139b4099d6" exitCode=0
Nov 03 21:29:52.574343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:52.573943    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j" event={"ID":"d8b774c1-1ed3-4a04-9861-244ba96d259c","Type":"ContainerDied","Data":"40fd00de6a396be35caf8a01a53cc2cf6640c1a557972fc535b6ee139b4099d6"}
Nov 03 21:29:53.742029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.742002    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:53.873943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.873859    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\" (UniqueName: \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\") pod \"d8b774c1-1ed3-4a04-9861-244ba96d259c\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") "
Nov 03 21:29:53.874092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.873957    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\") pod \"d8b774c1-1ed3-4a04-9861-244ba96d259c\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") "
Nov 03 21:29:53.874092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.874034    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\") pod \"d8b774c1-1ed3-4a04-9861-244ba96d259c\" (UID: \"d8b774c1-1ed3-4a04-9861-244ba96d259c\") "
Nov 03 21:29:53.874475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.874447    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca" pod "d8b774c1-1ed3-4a04-9861-244ba96d259c" (UID: "d8b774c1-1ed3-4a04-9861-244ba96d259c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:29:53.876458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.876425    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp" pod "d8b774c1-1ed3-4a04-9861-244ba96d259c" (UID: "d8b774c1-1ed3-4a04-9861-244ba96d259c"). InnerVolumeSpecName "kube-api-access-qjhtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:29:53.877029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.877002    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca" pod "d8b774c1-1ed3-4a04-9861-244ba96d259c" (UID: "d8b774c1-1ed3-4a04-9861-244ba96d259c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:29:53.975403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.975372    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:53.975403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.975398    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhtp\" (UniqueName: \"kubernetes.io/projected/d8b774c1-1ed3-4a04-9861-244ba96d259c-kube-api-access-qjhtp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:53.975403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:53.975409    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b774c1-1ed3-4a04-9861-244ba96d259c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:29:54.591514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:54.591482    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"
Nov 03 21:29:54.591514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:54.591506    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j" event={"ID":"d8b774c1-1ed3-4a04-9861-244ba96d259c","Type":"ContainerDied","Data":"26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737"}
Nov 03 21:29:54.591756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:54.591557    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c5f7041ac1166c54ec1abfbc7e2a7f3f5d05946974e3bebedd9309fbb53737"
Nov 03 21:29:57.448012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.447963    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:29:57.448738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.448710    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d8b774c1-1ed3-4a04-9861-244ba96d259c" containerName="revoke-keys"
Nov 03 21:29:57.448738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.448734    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b774c1-1ed3-4a04-9861-244ba96d259c" containerName="revoke-keys"
Nov 03 21:29:57.448915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.448892    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b774c1-1ed3-4a04-9861-244ba96d259c" containerName="revoke-keys"
Nov 03 21:29:57.452704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.452683    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.486495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.486459    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:29:57.490667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.490648    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.513364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513339    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513390    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513480    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513529    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513586    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513629    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513752    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513779    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sx2\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.513892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.513808    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.533562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.533540    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 21:29:57.533834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.533815    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 21:29:57.542686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.542664    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:29:57.590383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.590355    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:29:57.614559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614537    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sx2\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.614722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614593    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.614722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614646    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7989s\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.614722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.614905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614774    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.614905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614811    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.614905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614867    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.615070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614906    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.614956    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615009    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615048    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615105    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.615290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615136    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.615290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615181    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615223    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.615290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615273    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615326    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.615562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615353    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.615726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.615684    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.616518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.616495    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.616893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.616868    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.618402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.618377    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.618509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.618447    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.621450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.621404    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.622142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.622100    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.698287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.697171    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 21:29:57.706574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.706543    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.716411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716344    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.716554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716417    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716524    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716558    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716587    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716659    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716718    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.716770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716752    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.717046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.716825    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7989s\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.719249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.719165    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.719744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.719599    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.720456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.720121    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.720456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.720408    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.722096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.722042    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sx2\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.727512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.727466    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.728517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.728495    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.734068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.734046    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.751876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.744596    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 21:29:57.762939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.762918    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9818d8\""
Nov 03 21:29:57.762939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.762918    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9818d8-dockercfg-bb9lm\""
Nov 03 21:29:57.769024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.769004    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:57.783517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.783493    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7989s\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.784367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.784343    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.803712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.803685    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:57.818668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.818590    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.818900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.818792    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57sm\" (UniqueName: \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.920923 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.920848    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.921095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.921050    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57sm\" (UniqueName: \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.922786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.922713    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:57.981004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:57.980942    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57sm\" (UniqueName: \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\") pod \"payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:58.069811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.069784    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:58.218728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.218608    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:29:58.218817 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:58.218788    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9387c8_4445_4592_899d_dea4674b0ab9.slice/crio-936ae052319e5fe69745fe8774f9341056e060719c40365ba8c73a42db42ab45 WatchSource:0}: Error finding container 936ae052319e5fe69745fe8774f9341056e060719c40365ba8c73a42db42ab45: Status 404 returned error can't find the container with id 936ae052319e5fe69745fe8774f9341056e060719c40365ba8c73a42db42ab45
Nov 03 21:29:58.221505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.221476    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.221600 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:29:58.221513    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20ceb59_6017_429b_a02b_63863f27d2d6.slice/crio-2f6fefb172c7eeda33e61e1b8505b8f27a568dd5995039b3ab87bbf5b13272a6 WatchSource:0}: Error finding container 2f6fefb172c7eeda33e61e1b8505b8f27a568dd5995039b3ab87bbf5b13272a6: Status 404 returned error can't find the container with id 2f6fefb172c7eeda33e61e1b8505b8f27a568dd5995039b3ab87bbf5b13272a6
Nov 03 21:29:58.221600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.221537    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.221600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.221567    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.223742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.223719    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:29:58.224574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.224457    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.224574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.224512    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.224574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.224539    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:29:58.496577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.487542    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 21:29:58.645721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.645595    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" event={"ID":"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9","Type":"ContainerStarted","Data":"def5c01fbb822b21c0ae4559cae5cb4318c2f2ca725acdc262336a8d31fa9602"}
Nov 03 21:29:58.660475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.659509    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" event={"ID":"f20ceb59-6017-429b-a02b-63863f27d2d6","Type":"ContainerStarted","Data":"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"}
Nov 03 21:29:58.660475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.659568    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" event={"ID":"f20ceb59-6017-429b-a02b-63863f27d2d6","Type":"ContainerStarted","Data":"2f6fefb172c7eeda33e61e1b8505b8f27a568dd5995039b3ab87bbf5b13272a6"}
Nov 03 21:29:58.663443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.662789    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" event={"ID":"5e9387c8-4445-4592-899d-dea4674b0ab9","Type":"ContainerStarted","Data":"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"}
Nov 03 21:29:58.663443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.662849    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" event={"ID":"5e9387c8-4445-4592-899d-dea4674b0ab9","Type":"ContainerStarted","Data":"936ae052319e5fe69745fe8774f9341056e060719c40365ba8c73a42db42ab45"}
Nov 03 21:29:58.785655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.770713    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:29:58.810011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.804743    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:29:58.839656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.832847    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podStartSLOduration=1.832826806 podStartE2EDuration="1.832826806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:58.793056789 +0000 UTC m=+2500.424370270" watchObservedRunningTime="2026-11-03 21:29:58.832826806 +0000 UTC m=+2500.464140283"
Nov 03 21:29:58.929127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:58.929071    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podStartSLOduration=1.929050173 podStartE2EDuration="1.929050173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:58.924715916 +0000 UTC m=+2500.556029394" watchObservedRunningTime="2026-11-03 21:29:58.929050173 +0000 UTC m=+2500.560363650"
Nov 03 21:29:59.783295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.783258    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:29:59.786707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.786595    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:29:59.816372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.816334    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:29:59.816662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.816612    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:29:59.819008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.818978    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" event={"ID":"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9","Type":"ContainerStarted","Data":"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"}
Nov 03 21:29:59.819802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.819782    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:29:59.935952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:29:59.935891    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" podStartSLOduration=2.935868226 podStartE2EDuration="2.935868226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:59.934013753 +0000 UTC m=+2501.565327231" watchObservedRunningTime="2026-11-03 21:29:59.935868226 +0000 UTC m=+2501.567181705"
Nov 03 21:30:00.244659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.237382    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:00.273678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.268966    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.347675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.340018    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:30:00.387660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.348734    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:00.440561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.417960    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.440561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.418041    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2bj\" (UniqueName: \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.440561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.418971    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.475657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.474975    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:00.530707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.529932    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgx2k\" (UniqueName: \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.530707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.530006    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.530707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.530043    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.530707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.530100    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2bj\" (UniqueName: \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.553689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.531373    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.616877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.613424    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:00.634878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.634834    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgx2k\" (UniqueName: \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.635191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.635167    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.698555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.675481    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2bj\" (UniqueName: \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:00.698555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.676278    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.722492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.714597    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgx2k\" (UniqueName: \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\") pod \"maas-api-key-cleanup-29895690-8jzmc\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.771659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.747221    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:00.820700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.815076    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:00.820700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.815230    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:00.820700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.817701    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:00.820700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.817862    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:00.924667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.918514    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:00.924667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:00.918600    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqvh\" (UniqueName: \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.012994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.012919    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:01.061226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.060243    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqvh\" (UniqueName: \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.061226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.060521    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.062073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.061747    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.068612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.068558    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:01.076757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.076731    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:01.080968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.077005    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.084198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.084177    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:01.087123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.084415    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.110360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.110334    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:01.115466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.110601    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.126377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.126351    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:01.157038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.157005    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.211761    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgss\" (UniqueName: \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.211838    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkr6\" (UniqueName: \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.211905    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.211952    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.212232    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.214675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.212378    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttbr\" (UniqueName: \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.341263 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.341163    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttbr\" (UniqueName: \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.341543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.341501    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgss\" (UniqueName: \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.363854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.363815    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkr6\" (UniqueName: \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.364148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.364132    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.364653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.364601    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.389960    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.373492    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.373522    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.390213    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.390244    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.384387    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqvh\" (UniqueName: \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.372505    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.393821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.391033    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.432928    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgss\" (UniqueName: \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\") pod \"maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.437661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.437176    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttbr\" (UniqueName: \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\") pod \"maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.439188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.438969    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:01.439321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.439220    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:01.441531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.441504    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkr6\" (UniqueName: \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.635085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.634997    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:01.635442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.635425    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:01.733000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.730595    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:01.820738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.812027    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:01.820738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.812083    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:01.820738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.819738    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:01.820738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:01.819802    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:02.035057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.035009    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerStarted","Data":"8869aa7293249914a136ff513d2151d83f0dda1c92104e1d4b72bbdbe56448ef"}
Nov 03 21:30:02.101573 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:30:02.101259    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13bbdf7_45f9_4804_a3cc_fdec1057e919.slice/crio-ba26d4a7077fc2084231b128ce1033db44161bcc2ffb0c8b016adde52de1551e WatchSource:0}: Error finding container ba26d4a7077fc2084231b128ce1033db44161bcc2ffb0c8b016adde52de1551e: Status 404 returned error can't find the container with id ba26d4a7077fc2084231b128ce1033db44161bcc2ffb0c8b016adde52de1551e
Nov 03 21:30:02.107996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.107623    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:02.226870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.226839    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:02.238136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.238113    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:02.324676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.324322    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:02.329810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.329718    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:02.784446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.784414    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:02.784690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.784661    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:02.816079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.816051    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:02.816565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.816534    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:02.982806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:02.978759    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerName="main" containerID="cri-o://5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782" gracePeriod=45
Nov 03 21:30:03.044967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.044915    5533 generic.go:356] "Generic (PLEG): container finished" podID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerID="5ec64d174aaf49fe78aee093efad90f176373a53481dd34c493119d4b738cd6e" exitCode=60
Nov 03 21:30:03.045053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.045025    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerDied","Data":"5ec64d174aaf49fe78aee093efad90f176373a53481dd34c493119d4b738cd6e"}
Nov 03 21:30:03.045320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.045299    5533 scope.go:122] "RemoveContainer" containerID="5ec64d174aaf49fe78aee093efad90f176373a53481dd34c493119d4b738cd6e"
Nov 03 21:30:03.046912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.046888    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerStarted","Data":"50a22972f40c2378878cab23cd59e6d1c993831a279c7c77cd855e553c3ed2a2"}
Nov 03 21:30:03.047018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.046941    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerStarted","Data":"26311053bb16f88239f0f99bdc7efac1e4a08baea43c3983065c298180c7188b"}
Nov 03 21:30:03.048886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.048863    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerStarted","Data":"8fe4cedc2987bc09f1140249bafd8bb97ece8456abdeb718d754064b382c7aad"}
Nov 03 21:30:03.048988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.048906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerStarted","Data":"ba26d4a7077fc2084231b128ce1033db44161bcc2ffb0c8b016adde52de1551e"}
Nov 03 21:30:03.077658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.066364    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerStarted","Data":"d12339e58251a0c0d4fe47d0c1f1d7a03032c56f25ec79495eab2a2e30b69351"}
Nov 03 21:30:03.077658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.066415    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerStarted","Data":"b61864f92f2c9e84700143e593bbfbc0e06666402bcc4ce1c3eada9334c27f8c"}
Nov 03 21:30:03.119333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.119188    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerStarted","Data":"2839e586fa696c582bdee696f8d1684a83adfe4ba1201a9eb953be811aaef578"}
Nov 03 21:30:03.119333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.119264    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerStarted","Data":"9a4af65c25060580e3f9e713fecd44814764a439edcccc0a8b5d75b66f4115fb"}
Nov 03 21:30:03.229683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.229057    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerName="main" probeResult="failure" output="Get \"https://10.133.0.185:8000/ready\": dial tcp 10.133.0.185:8000: connect: connection refused"
Nov 03 21:30:03.240162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.239279    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerStarted","Data":"39af9ea192156fbbbd342e0d9b7b752b15bbc599dc28fd4525c7a5bf3ce0e566"}
Nov 03 21:30:03.240162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.239344    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerStarted","Data":"a1764787a1e6a7014a1f7db649df1e8652914ab36594a81c1ae319303a239ad1"}
Nov 03 21:30:03.306974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.306855    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" podStartSLOduration=3.306834738 podStartE2EDuration="3.306834738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.292521798 +0000 UTC m=+2504.923835277" watchObservedRunningTime="2026-11-03 21:30:03.306834738 +0000 UTC m=+2504.938148215"
Nov 03 21:30:03.428030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.427953    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" podStartSLOduration=3.427930912 podStartE2EDuration="3.427930912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.410545594 +0000 UTC m=+2505.041859071" watchObservedRunningTime="2026-11-03 21:30:03.427930912 +0000 UTC m=+2505.059244390"
Nov 03 21:30:03.534788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.534738    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" podStartSLOduration=3.534724587 podStartE2EDuration="3.534724587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.529460714 +0000 UTC m=+2505.160774193" watchObservedRunningTime="2026-11-03 21:30:03.534724587 +0000 UTC m=+2505.166038071"
Nov 03 21:30:03.723054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.722991    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" podStartSLOduration=3.722973595 podStartE2EDuration="3.722973595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.713090959 +0000 UTC m=+2505.344404441" watchObservedRunningTime="2026-11-03 21:30:03.722973595 +0000 UTC m=+2505.354287073"
Nov 03 21:30:03.770041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.769931    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:03.770041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.769987    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:03.807086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.806957    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:03.807086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.807013    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:03.818928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.818789    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:30:03.827262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.827220    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" podStartSLOduration=3.827206066 podStartE2EDuration="3.827206066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:03.824921752 +0000 UTC m=+2505.456235231" watchObservedRunningTime="2026-11-03 21:30:03.827206066 +0000 UTC m=+2505.458519543"
Nov 03 21:30:03.918742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918706    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.918742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918740    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.918991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918837    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\" (UniqueName: \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.918991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918877    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.918991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918908    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.918991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.918975    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\" (UniqueName: \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\") pod \"99fb02a4-188f-4f84-a974-b62c5ed80653\" (UID: \"99fb02a4-188f-4f84-a974-b62c5ed80653\") "
Nov 03 21:30:03.919239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919024    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:03.919239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919152    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:03.919239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919181    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:03.919471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919441    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:03.919544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919482    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:03.919544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.919499    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:03.921146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.921124    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:03.921411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.921394    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:03.921496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:03.921426    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w" pod "99fb02a4-188f-4f84-a974-b62c5ed80653" (UID: "99fb02a4-188f-4f84-a974-b62c5ed80653"). InnerVolumeSpecName "kube-api-access-l8g5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:04.041109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.041024    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fb02a4-188f-4f84-a974-b62c5ed80653-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:04.041109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.041060    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99fb02a4-188f-4f84-a974-b62c5ed80653-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:04.041109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.041076    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8g5w\" (UniqueName: \"kubernetes.io/projected/99fb02a4-188f-4f84-a974-b62c5ed80653-kube-api-access-l8g5w\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:04.310781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.301746    5533 generic.go:356] "Generic (PLEG): container finished" podID="47aab7f8-076a-4995-a657-25e3d104708f" containerID="39af9ea192156fbbbd342e0d9b7b752b15bbc599dc28fd4525c7a5bf3ce0e566" exitCode=60
Nov 03 21:30:04.310781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.301964    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerDied","Data":"39af9ea192156fbbbd342e0d9b7b752b15bbc599dc28fd4525c7a5bf3ce0e566"}
Nov 03 21:30:04.310781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.302419    5533 scope.go:122] "RemoveContainer" containerID="39af9ea192156fbbbd342e0d9b7b752b15bbc599dc28fd4525c7a5bf3ce0e566"
Nov 03 21:30:04.317935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.317420    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerStarted","Data":"ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"}
Nov 03 21:30:04.319804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.319764    5533 generic.go:356] "Generic (PLEG): container finished" podID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerID="5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782" exitCode=0
Nov 03 21:30:04.319951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.319933    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"
Nov 03 21:30:04.320462 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.320441    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" event={"ID":"99fb02a4-188f-4f84-a974-b62c5ed80653","Type":"ContainerDied","Data":"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"}
Nov 03 21:30:04.320582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.320481    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb" event={"ID":"99fb02a4-188f-4f84-a974-b62c5ed80653","Type":"ContainerDied","Data":"6a134ad07caa0e9585f8a9cc860f74736319900b61958bb344a80865189a82bd"}
Nov 03 21:30:04.320582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.320506    5533 scope.go:122] "RemoveContainer" containerID="5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"
Nov 03 21:30:04.322247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.322225    5533 generic.go:356] "Generic (PLEG): container finished" podID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerID="50a22972f40c2378878cab23cd59e6d1c993831a279c7c77cd855e553c3ed2a2" exitCode=60
Nov 03 21:30:04.322339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.322276    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerDied","Data":"50a22972f40c2378878cab23cd59e6d1c993831a279c7c77cd855e553c3ed2a2"}
Nov 03 21:30:04.322553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.322539    5533 scope.go:122] "RemoveContainer" containerID="50a22972f40c2378878cab23cd59e6d1c993831a279c7c77cd855e553c3ed2a2"
Nov 03 21:30:04.325469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.325452    5533 generic.go:356] "Generic (PLEG): container finished" podID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerID="8fe4cedc2987bc09f1140249bafd8bb97ece8456abdeb718d754064b382c7aad" exitCode=60
Nov 03 21:30:04.325704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.325664    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerDied","Data":"8fe4cedc2987bc09f1140249bafd8bb97ece8456abdeb718d754064b382c7aad"}
Nov 03 21:30:04.325793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.325766    5533 scope.go:122] "RemoveContainer" containerID="8fe4cedc2987bc09f1140249bafd8bb97ece8456abdeb718d754064b382c7aad"
Nov 03 21:30:04.328167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.328145    5533 generic.go:356] "Generic (PLEG): container finished" podID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerID="d12339e58251a0c0d4fe47d0c1f1d7a03032c56f25ec79495eab2a2e30b69351" exitCode=60
Nov 03 21:30:04.328295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.328276    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerDied","Data":"d12339e58251a0c0d4fe47d0c1f1d7a03032c56f25ec79495eab2a2e30b69351"}
Nov 03 21:30:04.328716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.328579    5533 scope.go:122] "RemoveContainer" containerID="d12339e58251a0c0d4fe47d0c1f1d7a03032c56f25ec79495eab2a2e30b69351"
Nov 03 21:30:04.329812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.329795    5533 generic.go:356] "Generic (PLEG): container finished" podID="9772b259-8219-4df2-bf18-70079abac44a" containerID="2839e586fa696c582bdee696f8d1684a83adfe4ba1201a9eb953be811aaef578" exitCode=60
Nov 03 21:30:04.329887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.329868    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerDied","Data":"2839e586fa696c582bdee696f8d1684a83adfe4ba1201a9eb953be811aaef578"}
Nov 03 21:30:04.330082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.330069    5533 scope.go:122] "RemoveContainer" containerID="2839e586fa696c582bdee696f8d1684a83adfe4ba1201a9eb953be811aaef578"
Nov 03 21:30:04.338198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.338177    5533 scope.go:122] "RemoveContainer" containerID="5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"
Nov 03 21:30:04.338495 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:04.338469    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782\": container with ID starting with 5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782 not found: ID does not exist" containerID="5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"
Nov 03 21:30:04.338537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.338510    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782"} err="failed to get container status \"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782\": rpc error: code = NotFound desc = could not find container \"5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782\": container with ID starting with 5069bccd0eef46c7b5175f105a1bbed68a10be5fb40c58bdfbe2216a0096d782 not found: ID does not exist"
Nov 03 21:30:04.695685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.687065    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" podStartSLOduration=4.68704511 podStartE2EDuration="4.68704511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:04.659587917 +0000 UTC m=+2506.290901395" watchObservedRunningTime="2026-11-03 21:30:04.68704511 +0000 UTC m=+2506.318358567"
Nov 03 21:30:04.793658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.776150    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:04.793658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.776206    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:04.811846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.809515    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:04.811846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.809589    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:04.821060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.821028    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:30:04.826125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:04.826099    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-3d317d/praxis-route-3d317d-kserve-85677854fc-t45hb"]
Nov 03 21:30:05.340702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.340587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerStarted","Data":"19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"}
Nov 03 21:30:05.342977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.342939    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerStarted","Data":"c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"}
Nov 03 21:30:05.345128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.345103    5533 generic.go:356] "Generic (PLEG): container finished" podID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerID="ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0" exitCode=60
Nov 03 21:30:05.345235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.345179    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerDied","Data":"ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"}
Nov 03 21:30:05.345297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.345234    5533 scope.go:122] "RemoveContainer" containerID="5ec64d174aaf49fe78aee093efad90f176373a53481dd34c493119d4b738cd6e"
Nov 03 21:30:05.345386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.345371    5533 scope.go:122] "RemoveContainer" containerID="ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"
Nov 03 21:30:05.345654 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:05.345617    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-8jzmc_odh-ai-gateway-infra(8aee11d2-b033-45fe-9ffa-6595cf78a36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a"
Nov 03 21:30:05.348737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.348715    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerStarted","Data":"8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"}
Nov 03 21:30:05.351209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.351188    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerStarted","Data":"10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"}
Nov 03 21:30:05.354122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.354099    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerStarted","Data":"ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"}
Nov 03 21:30:05.769714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.769684    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 21:30:05.769829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.769742    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 21:30:05.804662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.804617    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 21:30:05.804779 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:05.804671    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 21:30:06.535354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.534478    5533 generic.go:356] "Generic (PLEG): container finished" podID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerID="ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe" exitCode=60
Nov 03 21:30:06.535354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.534941    5533 scope.go:122] "RemoveContainer" containerID="ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"
Nov 03 21:30:06.535354 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.535217    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8_odh-ai-gateway-infra(08f2c368-86b8-4f8b-b73a-01f42fbe0738)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738"
Nov 03 21:30:06.540112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.540084    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" path="/var/lib/kubelet/pods/99fb02a4-188f-4f84-a974-b62c5ed80653/volumes"
Nov 03 21:30:06.540754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.540704    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerDied","Data":"ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"}
Nov 03 21:30:06.540989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.540968    5533 scope.go:122] "RemoveContainer" containerID="d12339e58251a0c0d4fe47d0c1f1d7a03032c56f25ec79495eab2a2e30b69351"
Nov 03 21:30:06.545581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.545240    5533 generic.go:356] "Generic (PLEG): container finished" podID="9772b259-8219-4df2-bf18-70079abac44a" containerID="19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1" exitCode=60
Nov 03 21:30:06.545581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.545390    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerDied","Data":"19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"}
Nov 03 21:30:06.549678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.545950    5533 scope.go:122] "RemoveContainer" containerID="19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"
Nov 03 21:30:06.549678 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.546217    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x_odh-ai-gateway-infra(9772b259-8219-4df2-bf18-70079abac44a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" podUID="9772b259-8219-4df2-bf18-70079abac44a"
Nov 03 21:30:06.563652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.563612    5533 generic.go:356] "Generic (PLEG): container finished" podID="47aab7f8-076a-4995-a657-25e3d104708f" containerID="c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e" exitCode=60
Nov 03 21:30:06.564006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.563982    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerDied","Data":"c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"}
Nov 03 21:30:06.564526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.564507    5533 scope.go:122] "RemoveContainer" containerID="c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"
Nov 03 21:30:06.564967 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.564942    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb_odh-ai-gateway-infra(47aab7f8-076a-4995-a657-25e3d104708f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" podUID="47aab7f8-076a-4995-a657-25e3d104708f"
Nov 03 21:30:06.572579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.572551    5533 scope.go:122] "RemoveContainer" containerID="ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"
Nov 03 21:30:06.572863 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.572829    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-8jzmc_odh-ai-gateway-infra(8aee11d2-b033-45fe-9ffa-6595cf78a36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a"
Nov 03 21:30:06.575573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.575547    5533 generic.go:356] "Generic (PLEG): container finished" podID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerID="8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897" exitCode=60
Nov 03 21:30:06.575823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.575624    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerDied","Data":"8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"}
Nov 03 21:30:06.575998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.575976    5533 scope.go:122] "RemoveContainer" containerID="8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"
Nov 03 21:30:06.576254 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.576228    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb_odh-ai-gateway-infra(496a4d8c-ca86-4291-a076-7fdd6294a6b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9"
Nov 03 21:30:06.583670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.580286    5533 generic.go:356] "Generic (PLEG): container finished" podID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerID="10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58" exitCode=60
Nov 03 21:30:06.583670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.580322    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerDied","Data":"10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"}
Nov 03 21:30:06.583670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.580386    5533 scope.go:122] "RemoveContainer" containerID="2839e586fa696c582bdee696f8d1684a83adfe4ba1201a9eb953be811aaef578"
Nov 03 21:30:06.583670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.580703    5533 scope.go:122] "RemoveContainer" containerID="10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"
Nov 03 21:30:06.583670 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:06.580922    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m_odh-ai-gateway-infra(f13bbdf7-45f9-4804-a3cc-fdec1057e919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919"
Nov 03 21:30:06.608239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.607501    5533 scope.go:122] "RemoveContainer" containerID="39af9ea192156fbbbd342e0d9b7b752b15bbc599dc28fd4525c7a5bf3ce0e566"
Nov 03 21:30:06.631357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.629135    5533 scope.go:122] "RemoveContainer" containerID="50a22972f40c2378878cab23cd59e6d1c993831a279c7c77cd855e553c3ed2a2"
Nov 03 21:30:06.650070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:06.649775    5533 scope.go:122] "RemoveContainer" containerID="8fe4cedc2987bc09f1140249bafd8bb97ece8456abdeb718d754064b382c7aad"
Nov 03 21:30:07.606050 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.605270    5533 scope.go:122] "RemoveContainer" containerID="19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"
Nov 03 21:30:07.606050 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:07.605629    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x_odh-ai-gateway-infra(9772b259-8219-4df2-bf18-70079abac44a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" podUID="9772b259-8219-4df2-bf18-70079abac44a"
Nov 03 21:30:07.610201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.610177    5533 scope.go:122] "RemoveContainer" containerID="c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"
Nov 03 21:30:07.610583 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:07.610558    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb_odh-ai-gateway-infra(47aab7f8-076a-4995-a657-25e3d104708f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" podUID="47aab7f8-076a-4995-a657-25e3d104708f"
Nov 03 21:30:07.613102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.613083    5533 scope.go:122] "RemoveContainer" containerID="8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"
Nov 03 21:30:07.613448 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:07.613424    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb_odh-ai-gateway-infra(496a4d8c-ca86-4291-a076-7fdd6294a6b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9"
Nov 03 21:30:07.616010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.615992    5533 scope.go:122] "RemoveContainer" containerID="10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"
Nov 03 21:30:07.616324 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:07.616302    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m_odh-ai-gateway-infra(f13bbdf7-45f9-4804-a3cc-fdec1057e919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919"
Nov 03 21:30:07.618929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.618907    5533 scope.go:122] "RemoveContainer" containerID="ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"
Nov 03 21:30:07.619159 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:07.619134    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8_odh-ai-gateway-infra(08f2c368-86b8-4f8b-b73a-01f42fbe0738)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738"
Nov 03 21:30:07.813659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.813587    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:30:07.813832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.813672    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:30:07.860710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.858710    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:30:07.860710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:07.858761    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:30:08.194657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.193678    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:30:08.206661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.195191    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:30:08.216658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.211975    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:30:08.338924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.338891    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:30:08.339308 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.339286    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:30:08.340021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:08.339999    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:30:09.207791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.207752    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:30:09.208553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.208531    5533 cpu_manager.go:414] "containerMap: removing container" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerName="main"
Nov 03 21:30:09.208677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.208555    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerName="main"
Nov 03 21:30:09.208749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.208718    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fb02a4-188f-4f84-a974-b62c5ed80653" containerName="main"
Nov 03 21:30:09.213474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.213454    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.271451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.271420    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:30:09.333322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333292    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333329    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333349    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333466    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333525    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333609    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333672    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nkp\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.333797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.333791    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.434721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434694    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.434854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434728    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.434854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434745    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.434854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434776    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.434854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434808    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434854    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434890    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nkp\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434927    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.434970    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.435135    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.435195    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.435281    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.435525    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.435686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.435531    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.437389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.437364    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.437768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.437747    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.492803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.492742    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.492915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.492803    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nkp\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\") pod \"e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") " pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.527711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.527689    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:09.957564 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:30:09.957537    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b3898b_a3a1_49a7_b753_3186a3b18414.slice/crio-58a26c8c7529b288e73a2c2718a5a016ab468fb52b64ea5ce12216e9b06dde80 WatchSource:0}: Error finding container 58a26c8c7529b288e73a2c2718a5a016ab468fb52b64ea5ce12216e9b06dde80: Status 404 returned error can't find the container with id 58a26c8c7529b288e73a2c2718a5a016ab468fb52b64ea5ce12216e9b06dde80
Nov 03 21:30:09.958862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.958836    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:30:09.959899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.959868    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:09.959961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.959936    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:09.960011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:09.959965    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:10.680457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:10.680417    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" event={"ID":"83b3898b-a3a1-49a7-b753-3186a3b18414","Type":"ContainerStarted","Data":"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"}
Nov 03 21:30:10.680945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:10.680481    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" event={"ID":"83b3898b-a3a1-49a7-b753-3186a3b18414","Type":"ContainerStarted","Data":"58a26c8c7529b288e73a2c2718a5a016ab468fb52b64ea5ce12216e9b06dde80"}
Nov 03 21:30:10.796595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:10.796540    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podStartSLOduration=1.796526651 podStartE2EDuration="1.796526651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:10.793068284 +0000 UTC m=+2512.424381760" watchObservedRunningTime="2026-11-03 21:30:10.796526651 +0000 UTC m=+2512.427840127"
Nov 03 21:30:11.528815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:11.528786    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:11.530018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:11.529992    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:11.530156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:11.530051    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:12.037611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.037581    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 21:30:12.528864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.528828    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:12.529041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.528880    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:12.591252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.591220    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:30:12.595746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.595720    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.686791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.686760    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.694893    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.694949    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695002    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695099    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695122    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695163    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695186    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695216    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.701561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.695257    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796129    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796167    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796194    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796231    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796299    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796323    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796363    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796406    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796431    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796555    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.796862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796755    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.797032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.796923    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.797032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.797002    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.797231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.797203    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.798773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.798751    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.798903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.798804    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.876897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.876864    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.877963 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.877936    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\") pod \"e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") " pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:12.909993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:12.909969    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:13.371658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.371141    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:30:13.391308 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:30:13.391282    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf504460b_0099_4ecb_a8f8_e3493bae4258.slice/crio-35e6a650592d94b203a801ecee47062b1d6401da0b81af639d46e67c05c191db WatchSource:0}: Error finding container 35e6a650592d94b203a801ecee47062b1d6401da0b81af639d46e67c05c191db: Status 404 returned error can't find the container with id 35e6a650592d94b203a801ecee47062b1d6401da0b81af639d46e67c05c191db
Nov 03 21:30:13.393872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.393839    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:13.393951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.393906    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:13.393951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.393932    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:30:13.528064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.528035    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:13.528181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.528082    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:13.720783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.720748    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" event={"ID":"f504460b-0099-4ecb-a8f8-e3493bae4258","Type":"ContainerStarted","Data":"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"}
Nov 03 21:30:13.720952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.720811    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" event={"ID":"f504460b-0099-4ecb-a8f8-e3493bae4258","Type":"ContainerStarted","Data":"35e6a650592d94b203a801ecee47062b1d6401da0b81af639d46e67c05c191db"}
Nov 03 21:30:13.879424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.879353    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podStartSLOduration=1.879329231 podStartE2EDuration="1.879329231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:13.84801817 +0000 UTC m=+2515.479331670" watchObservedRunningTime="2026-11-03 21:30:13.879329231 +0000 UTC m=+2515.510642709"
Nov 03 21:30:13.911077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:13.911029    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:14.528113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:14.528081    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:14.528509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:14.528131    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:14.913826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:14.913730    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:14.914101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:14.914069    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:15.033212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.033172    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 21:30:15.037755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.037730    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.107496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.107461    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 21:30:15.141976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.141934    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.142169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.142017    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.147627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.146952    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-c11538-dockercfg-dpffn\""
Nov 03 21:30:15.147627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.147051    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-c11538\""
Nov 03 21:30:15.340150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.339682    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.340150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.340001    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.340824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.340798    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.408401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.408334    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\") pod \"payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.528266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.528233    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:15.528664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.528280    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:15.670708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.669829    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:15.911395 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.911356    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:15.911728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:15.911692    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:16.137603 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:30:16.137572    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eed1179_c83c_4492_86e5_3388c404067a.slice/crio-166bbd5ad456427afdc76fd75ee8d1c223909e9878951b7008ec11c4e639856a WatchSource:0}: Error finding container 166bbd5ad456427afdc76fd75ee8d1c223909e9878951b7008ec11c4e639856a: Status 404 returned error can't find the container with id 166bbd5ad456427afdc76fd75ee8d1c223909e9878951b7008ec11c4e639856a
Nov 03 21:30:16.141027 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.140958    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 21:30:16.487466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.487433    5533 scope.go:122] "RemoveContainer" containerID="19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"
Nov 03 21:30:16.528671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.528622    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 21:30:16.528977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.528698    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 21:30:16.790206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.790168    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" event={"ID":"2eed1179-c83c-4492-86e5-3388c404067a","Type":"ContainerStarted","Data":"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"}
Nov 03 21:30:16.790364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.790235    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" event={"ID":"2eed1179-c83c-4492-86e5-3388c404067a","Type":"ContainerStarted","Data":"166bbd5ad456427afdc76fd75ee8d1c223909e9878951b7008ec11c4e639856a"}
Nov 03 21:30:16.791145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.791121    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:16.901888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.901832    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" podStartSLOduration=2.901814396 podStartE2EDuration="2.901814396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:16.899650001 +0000 UTC m=+2518.530963473" watchObservedRunningTime="2026-11-03 21:30:16.901814396 +0000 UTC m=+2518.533127876"
Nov 03 21:30:16.911011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.910984    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:16.911121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:16.911046    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:17.505931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.487464    5533 scope.go:122] "RemoveContainer" containerID="c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"
Nov 03 21:30:17.505931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.488576    5533 scope.go:122] "RemoveContainer" containerID="ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"
Nov 03 21:30:17.505931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.490540    5533 scope.go:122] "RemoveContainer" containerID="ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"
Nov 03 21:30:17.615379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.615336    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:30:17.622109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.622087    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.633503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.633476    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:30:17.709658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.692267    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cc4de5\""
Nov 03 21:30:17.803658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.803053    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cc4de5-dockercfg-52s5v\""
Nov 03 21:30:17.809830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.809602    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.845659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.823695    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974nd\" (UniqueName: \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.856727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.855536    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:30:17.912016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.911064    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:17.912016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.911115    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:17.923608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.923033    5533 generic.go:356] "Generic (PLEG): container finished" podID="9772b259-8219-4df2-bf18-70079abac44a" containerID="870d02d7d6a2e949a6afd1d45f312b8416571275fcf493fc6615a579353edc85" exitCode=60
Nov 03 21:30:17.924689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.924617    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.924888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.924868    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974nd\" (UniqueName: \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.926135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.926112    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.926876    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerDied","Data":"870d02d7d6a2e949a6afd1d45f312b8416571275fcf493fc6615a579353edc85"}
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.926940    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.926978    5533 scope.go:122] "RemoveContainer" containerID="19b0ef17c509c3d2956bc86827efffeaa3cd1d9d1a0c5e30199dad33b5eedab1"
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.927072    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:17.927409    5533 scope.go:122] "RemoveContainer" containerID="870d02d7d6a2e949a6afd1d45f312b8416571275fcf493fc6615a579353edc85"
Nov 03 21:30:17.927711 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:17.927661    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x_odh-ai-gateway-infra(9772b259-8219-4df2-bf18-70079abac44a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" podUID="9772b259-8219-4df2-bf18-70079abac44a"
Nov 03 21:30:18.017916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.017825    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:18.018913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.018663    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:18.019590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.019543    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:30:18.026301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.026260    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.026412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.026386    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvpn\" (UniqueName: \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.104686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.102321    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974nd\" (UniqueName: \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\") pod \"payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:18.127689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.127625    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.127815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.127756    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvpn\" (UniqueName: \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.128787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.128765    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.188502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.188221    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvpn\" (UniqueName: \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\") pod \"payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.223599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.223568    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:18.259627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.258535    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:18.496575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.495758    5533 scope.go:122] "RemoveContainer" containerID="10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"
Nov 03 21:30:18.629317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.629288    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:30:18.726789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.726756    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:30:18.910598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.910572    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:18.910717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.910612    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:18.932931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.932906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerStarted","Data":"86a5f61479f9c8bb4dade04541e1b8b1e24158415e767e2900939bfdce09235f"}
Nov 03 21:30:18.934870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.934845    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerStarted","Data":"8282c7ab3fd5a68d767b5fa3ac3bb93644be65ea31076d201158ca1b3534da62"}
Nov 03 21:30:18.937951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.936896    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" event={"ID":"7989c489-429e-4fd5-b201-7184b8d23c84","Type":"ContainerStarted","Data":"f2bc3efc9f0b29c22660f8d6bcce6b0614fc25157759a0f12d63760729430bca"}
Nov 03 21:30:18.939942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.939907    5533 generic.go:356] "Generic (PLEG): container finished" podID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerID="7b27b636e187a7904348e07816d252ff41768c97f7577496c96eebf43944508b" exitCode=60
Nov 03 21:30:18.940039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.940000    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerDied","Data":"7b27b636e187a7904348e07816d252ff41768c97f7577496c96eebf43944508b"}
Nov 03 21:30:18.940079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.940042    5533 scope.go:122] "RemoveContainer" containerID="ec968a8ec30ad71d7d608eb2a002b72c454acd760a0783b3ab923a562f5df0fe"
Nov 03 21:30:18.940355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.940342    5533 scope.go:122] "RemoveContainer" containerID="7b27b636e187a7904348e07816d252ff41768c97f7577496c96eebf43944508b"
Nov 03 21:30:18.940601 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:18.940579    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8_odh-ai-gateway-infra(08f2c368-86b8-4f8b-b73a-01f42fbe0738)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738"
Nov 03 21:30:18.973752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.972023    5533 generic.go:356] "Generic (PLEG): container finished" podID="47aab7f8-076a-4995-a657-25e3d104708f" containerID="ad91f06ab92c01bac188168cd582138c61e3d694586566a49c9de37c5c8f22ad" exitCode=60
Nov 03 21:30:18.973752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.972184    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerDied","Data":"ad91f06ab92c01bac188168cd582138c61e3d694586566a49c9de37c5c8f22ad"}
Nov 03 21:30:18.973752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.972728    5533 scope.go:122] "RemoveContainer" containerID="ad91f06ab92c01bac188168cd582138c61e3d694586566a49c9de37c5c8f22ad"
Nov 03 21:30:18.973752 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:18.973208    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb_odh-ai-gateway-infra(47aab7f8-076a-4995-a657-25e3d104708f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" podUID="47aab7f8-076a-4995-a657-25e3d104708f"
Nov 03 21:30:18.981454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.981419    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" event={"ID":"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78","Type":"ContainerStarted","Data":"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"}
Nov 03 21:30:18.981658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.981483    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" event={"ID":"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78","Type":"ContainerStarted","Data":"04ec21fc9d25059f35db2948f4a3e09e0704af75aff0b2dc77059d6915ff1edb"}
Nov 03 21:30:18.985472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:18.983182    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:19.056478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.056445    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:19.091914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.089286    5533 scope.go:122] "RemoveContainer" containerID="c3081aeef49ebd9f969aeb7523f9a07c2bca36de86ab984cdeab7908b5e1be9e"
Nov 03 21:30:19.243547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.243524    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:19.285040 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:19.285006    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:30:19.346767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.346734    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\" (UniqueName: \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\") pod \"9772b259-8219-4df2-bf18-70079abac44a\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") "
Nov 03 21:30:19.346912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.346857    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\") pod \"9772b259-8219-4df2-bf18-70079abac44a\" (UID: \"9772b259-8219-4df2-bf18-70079abac44a\") "
Nov 03 21:30:19.347186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.347163    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca" pod "9772b259-8219-4df2-bf18-70079abac44a" (UID: "9772b259-8219-4df2-bf18-70079abac44a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:19.347303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.347289    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9772b259-8219-4df2-bf18-70079abac44a-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:19.348904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.348879    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr" pod "9772b259-8219-4df2-bf18-70079abac44a" (UID: "9772b259-8219-4df2-bf18-70079abac44a"). InnerVolumeSpecName "kube-api-access-gttbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:19.391866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.391811    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" podStartSLOduration=2.391796668 podStartE2EDuration="2.391796668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:19.389150109 +0000 UTC m=+2521.020463590" watchObservedRunningTime="2026-11-03 21:30:19.391796668 +0000 UTC m=+2521.023110144"
Nov 03 21:30:19.448796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.448767    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttbr\" (UniqueName: \"kubernetes.io/projected/9772b259-8219-4df2-bf18-70079abac44a-kube-api-access-gttbr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:19.486388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.486323    5533 scope.go:122] "RemoveContainer" containerID="8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"
Nov 03 21:30:19.910428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.910389    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 21:30:19.910929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:19.910453    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 21:30:20.000401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.000363    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerStarted","Data":"e9701678e6531ed90c7dc123d4e1aabfbe5766f9c1b6508d9824962fce62d8e2"}
Nov 03 21:30:20.018472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.018444    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"
Nov 03 21:30:20.018946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.018522    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x" event={"ID":"9772b259-8219-4df2-bf18-70079abac44a","Type":"ContainerDied","Data":"9a4af65c25060580e3f9e713fecd44814764a439edcccc0a8b5d75b66f4115fb"}
Nov 03 21:30:20.019064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.018994    5533 scope.go:122] "RemoveContainer" containerID="870d02d7d6a2e949a6afd1d45f312b8416571275fcf493fc6615a579353edc85"
Nov 03 21:30:20.024088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.024066    5533 generic.go:356] "Generic (PLEG): container finished" podID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerID="86a5f61479f9c8bb4dade04541e1b8b1e24158415e767e2900939bfdce09235f" exitCode=60
Nov 03 21:30:20.024209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.024174    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerDied","Data":"86a5f61479f9c8bb4dade04541e1b8b1e24158415e767e2900939bfdce09235f"}
Nov 03 21:30:20.024532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.024514    5533 scope.go:122] "RemoveContainer" containerID="86a5f61479f9c8bb4dade04541e1b8b1e24158415e767e2900939bfdce09235f"
Nov 03 21:30:20.024816 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:20.024796    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-8jzmc_odh-ai-gateway-infra(8aee11d2-b033-45fe-9ffa-6595cf78a36a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a"
Nov 03 21:30:20.027010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.026991    5533 generic.go:356] "Generic (PLEG): container finished" podID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerID="8282c7ab3fd5a68d767b5fa3ac3bb93644be65ea31076d201158ca1b3534da62" exitCode=60
Nov 03 21:30:20.027210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.027193    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerDied","Data":"8282c7ab3fd5a68d767b5fa3ac3bb93644be65ea31076d201158ca1b3534da62"}
Nov 03 21:30:20.027691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.027673    5533 scope.go:122] "RemoveContainer" containerID="8282c7ab3fd5a68d767b5fa3ac3bb93644be65ea31076d201158ca1b3534da62"
Nov 03 21:30:20.028088 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:20.028064    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m_odh-ai-gateway-infra(f13bbdf7-45f9-4804-a3cc-fdec1057e919)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919"
Nov 03 21:30:20.030238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.030213    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" event={"ID":"7989c489-429e-4fd5-b201-7184b8d23c84","Type":"ContainerStarted","Data":"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"}
Nov 03 21:30:20.030328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.030264    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:20.033033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.033008    5533 scope.go:122] "RemoveContainer" containerID="ae0386e5c4857bdb96fdfc607cd5eb348fa74f17669031d0562d93de04ebdfa0"
Nov 03 21:30:20.049225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.049205    5533 scope.go:122] "RemoveContainer" containerID="10905517ed61f82a4d37de938c6f336689921aee654796d3c008946b61e1cd58"
Nov 03 21:30:20.061775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.061747    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:20.166285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.166208    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:20.295715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.295526    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:20.344445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.344413    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:20.460806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.452235    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:20.460806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.459260    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-fe2df9-29895690-bwz8x"]
Nov 03 21:30:20.470000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.469966    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\" (UniqueName: \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\") pod \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") "
Nov 03 21:30:20.470129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.470029    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\") pod \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\" (UID: \"08f2c368-86b8-4f8b-b73a-01f42fbe0738\") "
Nov 03 21:30:20.471014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.470986    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca" pod "08f2c368-86b8-4f8b-b73a-01f42fbe0738" (UID: "08f2c368-86b8-4f8b-b73a-01f42fbe0738"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:20.473016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.472963    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6" pod "08f2c368-86b8-4f8b-b73a-01f42fbe0738" (UID: "08f2c368-86b8-4f8b-b73a-01f42fbe0738"). InnerVolumeSpecName "kube-api-access-pmkr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:20.485495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.485473    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:20.497898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.497872    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9772b259-8219-4df2-bf18-70079abac44a" path="/var/lib/kubelet/pods/9772b259-8219-4df2-bf18-70079abac44a/volumes"
Nov 03 21:30:20.511660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.511625    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:20.560496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.560426    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" podStartSLOduration=3.560411844 podStartE2EDuration="3.560411844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:20.556290001 +0000 UTC m=+2522.187603479" watchObservedRunningTime="2026-11-03 21:30:20.560411844 +0000 UTC m=+2522.191725320"
Nov 03 21:30:20.571296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.571267    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\" (UniqueName: \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\") pod \"47aab7f8-076a-4995-a657-25e3d104708f\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") "
Nov 03 21:30:20.571441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.571356    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\") pod \"47aab7f8-076a-4995-a657-25e3d104708f\" (UID: \"47aab7f8-076a-4995-a657-25e3d104708f\") "
Nov 03 21:30:20.572063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.572033    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca" pod "47aab7f8-076a-4995-a657-25e3d104708f" (UID: "47aab7f8-076a-4995-a657-25e3d104708f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:20.572805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.572588    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkr6\" (UniqueName: \"kubernetes.io/projected/08f2c368-86b8-4f8b-b73a-01f42fbe0738-kube-api-access-pmkr6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:20.572805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.572614    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f2c368-86b8-4f8b-b73a-01f42fbe0738-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:20.572805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.572629    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47aab7f8-076a-4995-a657-25e3d104708f-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:20.573984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.573954    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss" pod "47aab7f8-076a-4995-a657-25e3d104708f" (UID: "47aab7f8-076a-4995-a657-25e3d104708f"). InnerVolumeSpecName "kube-api-access-pzgss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:20.678701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:20.673889    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzgss\" (UniqueName: \"kubernetes.io/projected/47aab7f8-076a-4995-a657-25e3d104708f-kube-api-access-pzgss\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:21.039959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.039929    5533 generic.go:356] "Generic (PLEG): container finished" podID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerID="e9701678e6531ed90c7dc123d4e1aabfbe5766f9c1b6508d9824962fce62d8e2" exitCode=60
Nov 03 21:30:21.040439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.040012    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerDied","Data":"e9701678e6531ed90c7dc123d4e1aabfbe5766f9c1b6508d9824962fce62d8e2"}
Nov 03 21:30:21.040439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.040095    5533 scope.go:122] "RemoveContainer" containerID="8afe969d6312cee28125ae8bccdc672a6a35955e22684b8dce22e4c92d9c3897"
Nov 03 21:30:21.040518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.040506    5533 scope.go:122] "RemoveContainer" containerID="e9701678e6531ed90c7dc123d4e1aabfbe5766f9c1b6508d9824962fce62d8e2"
Nov 03 21:30:21.040905 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:21.040877    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb_odh-ai-gateway-infra(496a4d8c-ca86-4291-a076-7fdd6294a6b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9"
Nov 03 21:30:21.041822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.041801    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"
Nov 03 21:30:21.041910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.041826    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8" event={"ID":"08f2c368-86b8-4f8b-b73a-01f42fbe0738","Type":"ContainerDied","Data":"b61864f92f2c9e84700143e593bbfbc0e06666402bcc4ce1c3eada9334c27f8c"}
Nov 03 21:30:21.044794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.044778    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"
Nov 03 21:30:21.044896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.044804    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb" event={"ID":"47aab7f8-076a-4995-a657-25e3d104708f","Type":"ContainerDied","Data":"a1764787a1e6a7014a1f7db649df1e8652914ab36594a81c1ae319303a239ad1"}
Nov 03 21:30:21.058280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.058176    5533 scope.go:122] "RemoveContainer" containerID="7b27b636e187a7904348e07816d252ff41768c97f7577496c96eebf43944508b"
Nov 03 21:30:21.077861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.075273    5533 scope.go:122] "RemoveContainer" containerID="ad91f06ab92c01bac188168cd582138c61e3d694586566a49c9de37c5c8f22ad"
Nov 03 21:30:21.136680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.130293    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:21.276516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.276484    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:21.285198 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.285152    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a41883-29895690-jvptb"]
Nov 03 21:30:21.372429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.372396    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:21.376269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.376242    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:21.410340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.410308    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:21.410787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.410758    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:21.411703 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.411683    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:30:21.499033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499003    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\") pod \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") "
Nov 03 21:30:21.499257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499111    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\" (UniqueName: \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\") pod \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\" (UID: \"8aee11d2-b033-45fe-9ffa-6595cf78a36a\") "
Nov 03 21:30:21.499257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499158    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\") pod \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") "
Nov 03 21:30:21.499407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499339    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\" (UniqueName: \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\") pod \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\" (UID: \"f13bbdf7-45f9-4804-a3cc-fdec1057e919\") "
Nov 03 21:30:21.499466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499404    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca" pod "8aee11d2-b033-45fe-9ffa-6595cf78a36a" (UID: "8aee11d2-b033-45fe-9ffa-6595cf78a36a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:21.499863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.499829    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca" pod "f13bbdf7-45f9-4804-a3cc-fdec1057e919" (UID: "f13bbdf7-45f9-4804-a3cc-fdec1057e919"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:21.500357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.500332    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aee11d2-b033-45fe-9ffa-6595cf78a36a-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:21.500476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.500359    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13bbdf7-45f9-4804-a3cc-fdec1057e919-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:21.501614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.501590    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k" pod "8aee11d2-b033-45fe-9ffa-6595cf78a36a" (UID: "8aee11d2-b033-45fe-9ffa-6595cf78a36a"). InnerVolumeSpecName "kube-api-access-jgx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:21.501913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.501896    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj" pod "f13bbdf7-45f9-4804-a3cc-fdec1057e919" (UID: "f13bbdf7-45f9-4804-a3cc-fdec1057e919"). InnerVolumeSpecName "kube-api-access-pl2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:21.601146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.601072    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:21.601534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.601505    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgx2k\" (UniqueName: \"kubernetes.io/projected/8aee11d2-b033-45fe-9ffa-6595cf78a36a-kube-api-access-jgx2k\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:21.601619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.601534    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2bj\" (UniqueName: \"kubernetes.io/projected/f13bbdf7-45f9-4804-a3cc-fdec1057e919-kube-api-access-pl2bj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:21.605410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:21.605389    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-320f60-29895690-wb9k8"]
Nov 03 21:30:22.061118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.061087    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"
Nov 03 21:30:22.061118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.061100    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc" event={"ID":"8aee11d2-b033-45fe-9ffa-6595cf78a36a","Type":"ContainerDied","Data":"8869aa7293249914a136ff513d2151d83f0dda1c92104e1d4b72bbdbe56448ef"}
Nov 03 21:30:22.061713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.061177    5533 scope.go:122] "RemoveContainer" containerID="86a5f61479f9c8bb4dade04541e1b8b1e24158415e767e2900939bfdce09235f"
Nov 03 21:30:22.062935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.062916    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"
Nov 03 21:30:22.063078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.062990    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m" event={"ID":"f13bbdf7-45f9-4804-a3cc-fdec1057e919","Type":"ContainerDied","Data":"ba26d4a7077fc2084231b128ce1033db44161bcc2ffb0c8b016adde52de1551e"}
Nov 03 21:30:22.074868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.074809    5533 scope.go:122] "RemoveContainer" containerID="8282c7ab3fd5a68d767b5fa3ac3bb93644be65ea31076d201158ca1b3534da62"
Nov 03 21:30:22.222319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.222287    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:22.228947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.228922    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895690-6fz9m"]
Nov 03 21:30:22.239709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.238387    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:22.312265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.312205    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\") pod \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") "
Nov 03 21:30:22.312376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.312316    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\" (UniqueName: \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\") pod \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\" (UID: \"496a4d8c-ca86-4291-a076-7fdd6294a6b9\") "
Nov 03 21:30:22.312543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.312523    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca" pod "496a4d8c-ca86-4291-a076-7fdd6294a6b9" (UID: "496a4d8c-ca86-4291-a076-7fdd6294a6b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:22.314326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.314304    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh" pod "496a4d8c-ca86-4291-a076-7fdd6294a6b9" (UID: "496a4d8c-ca86-4291-a076-7fdd6294a6b9"). InnerVolumeSpecName "kube-api-access-cvqvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:22.413677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.413627    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496a4d8c-ca86-4291-a076-7fdd6294a6b9-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:22.413677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.413681    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqvh\" (UniqueName: \"kubernetes.io/projected/496a4d8c-ca86-4291-a076-7fdd6294a6b9-kube-api-access-cvqvh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:22.433546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.433520    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:22.437523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.437502    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-8jzmc"]
Nov 03 21:30:22.494301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.494274    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" path="/var/lib/kubelet/pods/08f2c368-86b8-4f8b-b73a-01f42fbe0738/volumes"
Nov 03 21:30:22.494789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.494773    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47aab7f8-076a-4995-a657-25e3d104708f" path="/var/lib/kubelet/pods/47aab7f8-076a-4995-a657-25e3d104708f/volumes"
Nov 03 21:30:22.495219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.495205    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" path="/var/lib/kubelet/pods/8aee11d2-b033-45fe-9ffa-6595cf78a36a/volumes"
Nov 03 21:30:22.495965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:22.495950    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" path="/var/lib/kubelet/pods/f13bbdf7-45f9-4804-a3cc-fdec1057e919/volumes"
Nov 03 21:30:23.079515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:23.079475    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb" event={"ID":"496a4d8c-ca86-4291-a076-7fdd6294a6b9","Type":"ContainerDied","Data":"26311053bb16f88239f0f99bdc7efac1e4a08baea43c3983065c298180c7188b"}
Nov 03 21:30:23.079924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:23.079552    5533 scope.go:122] "RemoveContainer" containerID="e9701678e6531ed90c7dc123d4e1aabfbe5766f9c1b6508d9824962fce62d8e2"
Nov 03 21:30:23.079924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:23.079692    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"
Nov 03 21:30:23.237765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:23.237740    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:23.241940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:23.241913    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895690-76tmb"]
Nov 03 21:30:24.496449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:24.496412    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" path="/var/lib/kubelet/pods/496a4d8c-ca86-4291-a076-7fdd6294a6b9/volumes"
Nov 03 21:30:25.111107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111073    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"]
Nov 03 21:30:25.111663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111620    5533 cpu_manager.go:414] "containerMap: removing container" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.111663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111647    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.111663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111658    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.111663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111662    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111672    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111678    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111686    5533 cpu_manager.go:414] "containerMap: removing container" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111691    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111708    5533 cpu_manager.go:414] "containerMap: removing container" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111712    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111724    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111729    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111736    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111740    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111747    5533 cpu_manager.go:414] "containerMap: removing container" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111751    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111757    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111762    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111771    5533 cpu_manager.go:414] "containerMap: removing container" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111777    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111795    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111802    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111812    5533 cpu_manager.go:414] "containerMap: removing container" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.111865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111817    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111914    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111923    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111931    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111937    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111945    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111953    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111958    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111963    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111969    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111976    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111980    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111985    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:30:25.112556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.111991    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:30:25.117226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.117208    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.171313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.171273    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"]
Nov 03 21:30:25.211829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.211801    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:30:25.211959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.211858    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:30:25.250185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.250162    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.250304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.250199    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchj8\" (UniqueName: \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.250304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.250238    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.351870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.351824    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.352079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.351873    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchj8\" (UniqueName: \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.352079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.351910    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.353196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.353159    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.357954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.357925    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.412323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.412108    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchj8\" (UniqueName: \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\") pod \"maas-api-revoke-sub-a54301df-6t729\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.431114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.431083    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:25.839936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:25.839899    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"]
Nov 03 21:30:25.840353 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:30:25.840237    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe913e4_7347_4c38_b7b6_1920c6c0ac06.slice/crio-62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1 WatchSource:0}: Error finding container 62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1: Status 404 returned error can't find the container with id 62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1
Nov 03 21:30:26.114258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:26.114168    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729" event={"ID":"afe913e4-7347-4c38-b7b6-1920c6c0ac06","Type":"ContainerStarted","Data":"62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1"}
Nov 03 21:30:26.379670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:26.379567    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:30:27.125872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:27.125837    5533 generic.go:356] "Generic (PLEG): container finished" podID="afe913e4-7347-4c38-b7b6-1920c6c0ac06" containerID="9a93d11928e38816146fe40ce8af8bd4d72088be074f23ef781f0ecc1b52652c" exitCode=0
Nov 03 21:30:27.126269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:27.125907    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729" event={"ID":"afe913e4-7347-4c38-b7b6-1920c6c0ac06","Type":"ContainerDied","Data":"9a93d11928e38816146fe40ce8af8bd4d72088be074f23ef781f0ecc1b52652c"}
Nov 03 21:30:29.008838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.008806    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:29.044316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.044286    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 21:30:29.094517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.094481    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\" (UniqueName: \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\") pod \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") "
Nov 03 21:30:29.094701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.094660    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\") pod \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") "
Nov 03 21:30:29.094769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.094744    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\") pod \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\" (UID: \"afe913e4-7347-4c38-b7b6-1920c6c0ac06\") "
Nov 03 21:30:29.095188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.095160    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca" pod "afe913e4-7347-4c38-b7b6-1920c6c0ac06" (UID: "afe913e4-7347-4c38-b7b6-1920c6c0ac06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:29.096763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.096740    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8" pod "afe913e4-7347-4c38-b7b6-1920c6c0ac06" (UID: "afe913e4-7347-4c38-b7b6-1920c6c0ac06"). InnerVolumeSpecName "kube-api-access-gchj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:29.097155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.097125    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca" pod "afe913e4-7347-4c38-b7b6-1920c6c0ac06" (UID: "afe913e4-7347-4c38-b7b6-1920c6c0ac06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:30:29.200470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200355    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchj8\" (UniqueName: \"kubernetes.io/projected/afe913e4-7347-4c38-b7b6-1920c6c0ac06-kube-api-access-gchj8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:29.200470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200390    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:29.200470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200404    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe913e4-7347-4c38-b7b6-1920c6c0ac06-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:29.200795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200652    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729" event={"ID":"afe913e4-7347-4c38-b7b6-1920c6c0ac06","Type":"ContainerDied","Data":"62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1"}
Nov 03 21:30:29.200795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200706    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c1e88004cd1c9c0ea8bc5c13b9b6e11b51ae7d4da847b6d6db9643ea30e9a1"
Nov 03 21:30:29.200795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:29.200769    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"
Nov 03 21:30:30.033393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:30.033364    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:30:31.053038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:31.053008    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:30:32.046599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:32.046569    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"]
Nov 03 21:30:32.093250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:32.093223    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3597afd9-nw7nf"]
Nov 03 21:30:32.494793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:32.494761    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1363be-79cd-4103-a588-5a87a285efb1" path="/var/lib/kubelet/pods/7c1363be-79cd-4103-a588-5a87a285efb1/volumes"
Nov 03 21:30:48.047552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:48.047520    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"]
Nov 03 21:30:48.094105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:48.094076    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec227956-xzxj7"]
Nov 03 21:30:48.496070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:48.496036    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b024a5b2-e2f6-40a4-afa2-bffbc8b44508" path="/var/lib/kubelet/pods/b024a5b2-e2f6-40a4-afa2-bffbc8b44508/volumes"
Nov 03 21:30:49.261909 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:49.261868    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:30:54.224914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:54.224875    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:30:54.225354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:54.225122    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy" containerID="cri-o://b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b" gracePeriod=30
Nov 03 21:30:59.514545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.514509    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:30:59.539562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.539521    5533 generic.go:356] "Generic (PLEG): container finished" podID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerID="b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b" exitCode=0
Nov 03 21:30:59.539788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.539580    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"
Nov 03 21:30:59.539788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.539657    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerDied","Data":"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"}
Nov 03 21:30:59.539788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.539727    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6" event={"ID":"ef42301c-9ba0-4ffd-83ce-14d30787e60d","Type":"ContainerDied","Data":"82e2fd19a8919de54bf498ab0eb8bda61e3af8345a18ff70b98db2d8d70f6836"}
Nov 03 21:30:59.539788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.539757    5533 scope.go:122] "RemoveContainer" containerID="b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"
Nov 03 21:30:59.551509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.551485    5533 scope.go:122] "RemoveContainer" containerID="c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"
Nov 03 21:30:59.563724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.563703    5533 scope.go:122] "RemoveContainer" containerID="b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"
Nov 03 21:30:59.564041 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:59.564017    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b\": container with ID starting with b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b not found: ID does not exist" containerID="b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"
Nov 03 21:30:59.564144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.564054    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b"} err="failed to get container status \"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b\": rpc error: code = NotFound desc = could not find container \"b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b\": container with ID starting with b6bb63dca69e142835d804edeaab0987863959bce446443b5405fbeec2cd7a6b not found: ID does not exist"
Nov 03 21:30:59.564144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.564079    5533 scope.go:122] "RemoveContainer" containerID="c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"
Nov 03 21:30:59.564408 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:30:59.564384    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe\": container with ID starting with c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe not found: ID does not exist" containerID="c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"
Nov 03 21:30:59.564473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.564410    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe"} err="failed to get container status \"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe\": rpc error: code = NotFound desc = could not find container \"c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe\": container with ID starting with c5f74999f4c5fc7fffaf7d48c0c7b1e52ac5e833f7e66b1e8d7da5d0fb93adbe not found: ID does not exist"
Nov 03 21:30:59.572801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.572773    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.572924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.572839    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.572924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.572873    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.572960    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.573044    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.573116    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.573280    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.573337    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.573694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.573409    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\") pod \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\" (UID: \"ef42301c-9ba0-4ffd-83ce-14d30787e60d\") "
Nov 03 21:30:59.574143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.574086    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:59.574560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.574509    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:59.575195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.575067    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.575195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.575097    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.576044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.575724    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:59.576346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.576322    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "kube-api-access-4wfqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:59.576491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.576398    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:59.577513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.577493    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:30:59.577786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.577762    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:59.579567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.579542    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:30:59.582232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.582201    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token" pod "ef42301c-9ba0-4ffd-83ce-14d30787e60d" (UID: "ef42301c-9ba0-4ffd-83ce-14d30787e60d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:59.676048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676012    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676045    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676061    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676075    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfqn\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-kube-api-access-4wfqn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676090    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676103    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef42301c-9ba0-4ffd-83ce-14d30787e60d-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.676227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.676117    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef42301c-9ba0-4ffd-83ce-14d30787e60d-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:59.962959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.962923    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:30:59.969060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:30:59.969030    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3d317d-openshift-default-69b5cdfc69-2b2h6"]
Nov 03 21:31:00.494304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:00.494273    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" path="/var/lib/kubelet/pods/ef42301c-9ba0-4ffd-83ce-14d30787e60d/volumes"
Nov 03 21:31:02.130655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:02.130608    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:31:02.184222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:02.184191    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:31:03.443606 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:03.443571    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:31:04.047810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:04.047778    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"]
Nov 03 21:31:04.093253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:04.093218    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f5c97-cm8vm"]
Nov 03 21:31:04.494154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:04.494122    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930a1dac-e875-4414-8390-f450a09dabc0" path="/var/lib/kubelet/pods/930a1dac-e875-4414-8390-f450a09dabc0/volumes"
Nov 03 21:31:07.131447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.131385    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" podUID="a662eef1-9966-4d85-99cb-ffb19ea06881" containerName="payload-processing" containerID="cri-o://66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34" gracePeriod=25
Nov 03 21:31:07.185264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.185228    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" podUID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" containerName="payload-pre-processing" containerID="cri-o://1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5" gracePeriod=25
Nov 03 21:31:07.456506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.456472    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:31:07.491832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.491806    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:31:07.582679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.582621    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\") pod \"a662eef1-9966-4d85-99cb-ffb19ea06881\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") "
Nov 03 21:31:07.582834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.582774    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\" (UniqueName: \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\") pod \"a662eef1-9966-4d85-99cb-ffb19ea06881\" (UID: \"a662eef1-9966-4d85-99cb-ffb19ea06881\") "
Nov 03 21:31:07.582834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.582819    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\" (UniqueName: \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\") pod \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") "
Nov 03 21:31:07.582953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.582906    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\") pod \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\" (UID: \"a0f81fc5-2722-4a35-9bfa-70ad571daee0\") "
Nov 03 21:31:07.583356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.582999    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume" pod "a662eef1-9966-4d85-99cb-ffb19ea06881" (UID: "a662eef1-9966-4d85-99cb-ffb19ea06881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:07.583510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.583417    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a662eef1-9966-4d85-99cb-ffb19ea06881-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:07.583832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.583807    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume" pod "a0f81fc5-2722-4a35-9bfa-70ad571daee0" (UID: "a0f81fc5-2722-4a35-9bfa-70ad571daee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:07.585915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.585884    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx" pod "a662eef1-9966-4d85-99cb-ffb19ea06881" (UID: "a662eef1-9966-4d85-99cb-ffb19ea06881"). InnerVolumeSpecName "kube-api-access-h5pbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:07.586285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.586267    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj" pod "a0f81fc5-2722-4a35-9bfa-70ad571daee0" (UID: "a0f81fc5-2722-4a35-9bfa-70ad571daee0"). InnerVolumeSpecName "kube-api-access-46krj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:07.606437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.606410    5533 generic.go:356] "Generic (PLEG): container finished" podID="a662eef1-9966-4d85-99cb-ffb19ea06881" containerID="66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34" exitCode=0
Nov 03 21:31:07.606561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.606458    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"
Nov 03 21:31:07.606561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.606501    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" event={"ID":"a662eef1-9966-4d85-99cb-ffb19ea06881","Type":"ContainerDied","Data":"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"}
Nov 03 21:31:07.606783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.606561    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj" event={"ID":"a662eef1-9966-4d85-99cb-ffb19ea06881","Type":"ContainerDied","Data":"83fe2c9063bb4297ca8276ac4ef99cd1c70a1151da714904571bb5f64ac9d518"}
Nov 03 21:31:07.606783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.606596    5533 scope.go:122] "RemoveContainer" containerID="66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"
Nov 03 21:31:07.608849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.608694    5533 generic.go:356] "Generic (PLEG): container finished" podID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" containerID="1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5" exitCode=0
Nov 03 21:31:07.608849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.608753    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"
Nov 03 21:31:07.609013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.608861    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" event={"ID":"a0f81fc5-2722-4a35-9bfa-70ad571daee0","Type":"ContainerDied","Data":"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"}
Nov 03 21:31:07.609013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.608913    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt" event={"ID":"a0f81fc5-2722-4a35-9bfa-70ad571daee0","Type":"ContainerDied","Data":"119b244000667dfa81b250c823e97325dbf0dba0242ced34e8004960cb817bb1"}
Nov 03 21:31:07.626113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.626025    5533 scope.go:122] "RemoveContainer" containerID="66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"
Nov 03 21:31:07.626406 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:31:07.626382    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34\": container with ID starting with 66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34 not found: ID does not exist" containerID="66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"
Nov 03 21:31:07.626511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.626414    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34"} err="failed to get container status \"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34\": rpc error: code = NotFound desc = could not find container \"66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34\": container with ID starting with 66d25c260b0b77483315d67d2e808e9648fbabe0ededc91dcdd1ee2ab9fcef34 not found: ID does not exist"
Nov 03 21:31:07.626511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.626440    5533 scope.go:122] "RemoveContainer" containerID="1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"
Nov 03 21:31:07.636391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.636371    5533 scope.go:122] "RemoveContainer" containerID="1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"
Nov 03 21:31:07.637298 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:31:07.637271    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5\": container with ID starting with 1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5 not found: ID does not exist" containerID="1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"
Nov 03 21:31:07.637381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.637307    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5"} err="failed to get container status \"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5\": rpc error: code = NotFound desc = could not find container \"1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5\": container with ID starting with 1b031bffa6e9242f1f12c92dd071646ef5d5c02e5e3203517f818d9ffd0acaf5 not found: ID does not exist"
Nov 03 21:31:07.685285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.685260    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5pbx\" (UniqueName: \"kubernetes.io/projected/a662eef1-9966-4d85-99cb-ffb19ea06881-kube-api-access-h5pbx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:07.685285 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.685284    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46krj\" (UniqueName: \"kubernetes.io/projected/a0f81fc5-2722-4a35-9bfa-70ad571daee0-kube-api-access-46krj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:07.685493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.685296    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f81fc5-2722-4a35-9bfa-70ad571daee0-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:07.788496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.788457    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:31:07.794671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.794625    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2df9-54f9486bf5-p74kt"]
Nov 03 21:31:07.895891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.895795    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:31:07.900649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:07.900616    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2df9-79b848fbf8-9f6zj"]
Nov 03 21:31:08.496279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:08.496243    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" path="/var/lib/kubelet/pods/a0f81fc5-2722-4a35-9bfa-70ad571daee0/volumes"
Nov 03 21:31:08.496810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:08.496789    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a662eef1-9966-4d85-99cb-ffb19ea06881" path="/var/lib/kubelet/pods/a662eef1-9966-4d85-99cb-ffb19ea06881/volumes"
Nov 03 21:31:19.261671 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:31:19.261616    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:31:21.057625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:21.057586    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"]
Nov 03 21:31:21.104166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:21.104134    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bfca6-c494c"]
Nov 03 21:31:22.495007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:22.494969    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189da769-4335-4ecc-a108-2a355ed33e5f" path="/var/lib/kubelet/pods/189da769-4335-4ecc-a108-2a355ed33e5f/volumes"
Nov 03 21:31:27.303394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.303357    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 21:31:27.304031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304014    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304032    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304046    5533 cpu_manager.go:414] "containerMap: removing container" podUID="afe913e4-7347-4c38-b7b6-1920c6c0ac06" containerName="revoke-keys"
Nov 03 21:31:27.304077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304052    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe913e4-7347-4c38-b7b6-1920c6c0ac06" containerName="revoke-keys"
Nov 03 21:31:27.304077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304069    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a662eef1-9966-4d85-99cb-ffb19ea06881" containerName="payload-processing"
Nov 03 21:31:27.304077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304075    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a662eef1-9966-4d85-99cb-ffb19ea06881" containerName="payload-processing"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304084    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304089    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304097    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304101    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304109    5533 cpu_manager.go:414] "containerMap: removing container" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304114    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304131    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304135    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304142    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" containerName="payload-pre-processing"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304147    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" containerName="payload-pre-processing"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304159    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304163    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304170    5533 cpu_manager.go:414] "containerMap: removing container" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304175    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496a4d8c-ca86-4291-a076-7fdd6294a6b9" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304180    5533 cpu_manager.go:414] "containerMap: removing container" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:31:27.304240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304184    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304255    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a662eef1-9966-4d85-99cb-ffb19ea06881" containerName="payload-processing"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304265    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f2c368-86b8-4f8b-b73a-01f42fbe0738" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304270    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13bbdf7-45f9-4804-a3cc-fdec1057e919" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304279    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304286    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f81fc5-2722-4a35-9bfa-70ad571daee0" containerName="payload-pre-processing"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304292    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aee11d2-b033-45fe-9ffa-6595cf78a36a" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304298    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef42301c-9ba0-4ffd-83ce-14d30787e60d" containerName="istio-proxy"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304302    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9772b259-8219-4df2-bf18-70079abac44a" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304307    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="47aab7f8-076a-4995-a657-25e3d104708f" containerName="cleanup"
Nov 03 21:31:27.304788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.304313    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe913e4-7347-4c38-b7b6-1920c6c0ac06" containerName="revoke-keys"
Nov 03 21:31:27.309450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.309432    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.452702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.452666    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6mr\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.453119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.453062    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.453375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.453340    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.453584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.453564    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.453887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.453871    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.454008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.453995    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.454122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.454108    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.454225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.454213    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.454353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.454336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.455195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.455171    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a1c634-openshift-default-dockercfg-54z6n\""
Nov 03 21:31:27.478627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.478484    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555737    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555785    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555819    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555850    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555887    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555952    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6mr\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.555980    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.556027    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.557742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.556093    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.561173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.561143    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.561422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.561401    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.563678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.563117    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.564363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.564201    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.564583 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.564556    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.564981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.564938    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.565133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.565113    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.621212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.621180    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.622330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.622301    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6mr\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\") pod \"e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") " pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:27.623012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:27.622990    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:28.055476 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:28.055446    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe62722_5163_49cb_9d8d_2704bace267e.slice/crio-267ecc30ff61530d3760d3f2cdbd1aa1c64efe45cc668b49363875ab557f140b WatchSource:0}: Error finding container 267ecc30ff61530d3760d3f2cdbd1aa1c64efe45cc668b49363875ab557f140b: Status 404 returned error can't find the container with id 267ecc30ff61530d3760d3f2cdbd1aa1c64efe45cc668b49363875ab557f140b
Nov 03 21:31:28.057689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.057648    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 21:31:28.058148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.058118    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:28.058231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.058181    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:28.058231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.058210    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:31:28.932681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.931526    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerStarted","Data":"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"}
Nov 03 21:31:28.932681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:28.931613    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerStarted","Data":"267ecc30ff61530d3760d3f2cdbd1aa1c64efe45cc668b49363875ab557f140b"}
Nov 03 21:31:29.053842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.053777    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podStartSLOduration=2.053757162 podStartE2EDuration="2.053757162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:29.05226755 +0000 UTC m=+2590.683581029" watchObservedRunningTime="2026-11-03 21:31:29.053757162 +0000 UTC m=+2590.685070639"
Nov 03 21:31:29.073626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.073590    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:31:29.623422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.623391    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:29.628776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.628748    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:29.938262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.938237    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:29.939084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:29.939054    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:31:31.048271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:31.048241    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"]
Nov 03 21:31:31.096025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:31.095993    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28f25e5-l9vz2"]
Nov 03 21:31:32.049795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.049766    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"]
Nov 03 21:31:32.096001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.095975    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdb9fd6-tlz4g"]
Nov 03 21:31:32.498898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.498868    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d40a61e-6aac-4691-9dc2-228902421d97" path="/var/lib/kubelet/pods/9d40a61e-6aac-4691-9dc2-228902421d97/volumes"
Nov 03 21:31:32.499428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.499411    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c083a0-c08a-48e4-8863-15c1c57e0f34" path="/var/lib/kubelet/pods/c1c083a0-c08a-48e4-8863-15c1c57e0f34/volumes"
Nov 03 21:31:32.893036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.892967    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"]
Nov 03 21:31:32.898204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.898185    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:32.948268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.948241    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:31:32.952155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.952130    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"]
Nov 03 21:31:32.957958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.957929    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:32.958124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.958105    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbq4\" (UniqueName: \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:32.958244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.958227    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:32.971892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:32.971868    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:31:33.059841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.059812    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.060271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.059860    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.060271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.060068    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbq4\" (UniqueName: \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.060724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.060701    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.062404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.062384    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.120422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.120399    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbq4\" (UniqueName: \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\") pod \"maas-api-revoke-sub-2a97d695-kgrd9\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.212433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.212409    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:33.583349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.583315    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"]
Nov 03 21:31:33.583731 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:33.583696    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7ca452_bdd6_44c4_8c3b_ec848842be91.slice/crio-20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5 WatchSource:0}: Error finding container 20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5: Status 404 returned error can't find the container with id 20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5
Nov 03 21:31:33.983177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.983142    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9" event={"ID":"8d7ca452-bdd6-44c4-8c3b-ec848842be91","Type":"ContainerStarted","Data":"60418aed24f9b97d4886ad33c8dd58b53ee437fcfa67cf71653306ecca502e11"}
Nov 03 21:31:33.983328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:33.983207    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9" event={"ID":"8d7ca452-bdd6-44c4-8c3b-ec848842be91","Type":"ContainerStarted","Data":"20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5"}
Nov 03 21:31:34.094922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:34.094873    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9" podStartSLOduration=2.094859485 podStartE2EDuration="2.094859485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:34.091631811 +0000 UTC m=+2595.722945287" watchObservedRunningTime="2026-11-03 21:31:34.094859485 +0000 UTC m=+2595.726172969"
Nov 03 21:31:34.991392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:34.991358    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d7ca452-bdd6-44c4-8c3b-ec848842be91" containerID="60418aed24f9b97d4886ad33c8dd58b53ee437fcfa67cf71653306ecca502e11" exitCode=0
Nov 03 21:31:34.991576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:34.991440    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9" event={"ID":"8d7ca452-bdd6-44c4-8c3b-ec848842be91","Type":"ContainerDied","Data":"60418aed24f9b97d4886ad33c8dd58b53ee437fcfa67cf71653306ecca502e11"}
Nov 03 21:31:36.144603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.144577    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:36.193742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.193717    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\") pod \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") "
Nov 03 21:31:36.193869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.193794    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\") pod \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") "
Nov 03 21:31:36.193869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.193857    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\" (UniqueName: \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\") pod \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\" (UID: \"8d7ca452-bdd6-44c4-8c3b-ec848842be91\") "
Nov 03 21:31:36.195866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.195835    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca" pod "8d7ca452-bdd6-44c4-8c3b-ec848842be91" (UID: "8d7ca452-bdd6-44c4-8c3b-ec848842be91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:36.196110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.196089    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca" pod "8d7ca452-bdd6-44c4-8c3b-ec848842be91" (UID: "8d7ca452-bdd6-44c4-8c3b-ec848842be91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:31:36.196233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.196205    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4" pod "8d7ca452-bdd6-44c4-8c3b-ec848842be91" (UID: "8d7ca452-bdd6-44c4-8c3b-ec848842be91"). InnerVolumeSpecName "kube-api-access-gqbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:36.295593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.295513    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:36.295593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.295554    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7ca452-bdd6-44c4-8c3b-ec848842be91-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:36.295593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:36.295571    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqbq4\" (UniqueName: \"kubernetes.io/projected/8d7ca452-bdd6-44c4-8c3b-ec848842be91-kube-api-access-gqbq4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:37.008898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:37.008868    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"
Nov 03 21:31:37.008898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:37.008887    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9" event={"ID":"8d7ca452-bdd6-44c4-8c3b-ec848842be91","Type":"ContainerDied","Data":"20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5"}
Nov 03 21:31:37.009159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:37.008942    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20291d323cb5d13085d9b44f04467ec288616d96898a6c0f1cc6e8a090e4fbb5"
Nov 03 21:31:39.585653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.584793    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 21:31:39.586048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.585664    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d7ca452-bdd6-44c4-8c3b-ec848842be91" containerName="revoke-keys"
Nov 03 21:31:39.586048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.585684    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7ca452-bdd6-44c4-8c3b-ec848842be91" containerName="revoke-keys"
Nov 03 21:31:39.586048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.585892    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7ca452-bdd6-44c4-8c3b-ec848842be91" containerName="revoke-keys"
Nov 03 21:31:39.590345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.590322    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.641885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.641859    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a1c634\""
Nov 03 21:31:39.668342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.668298    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wtjr\" (UniqueName: \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.668520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.668408    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.696187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.696156    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 21:31:39.702874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.702848    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 21:31:39.714363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.714329    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:39.784659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.783429    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtjr\" (UniqueName: \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.784659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.783546    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.784918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.784828    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.819692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.807719    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a1c634-dockercfg-dwr2d\""
Nov 03 21:31:39.848728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.845319    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 21:31:39.879658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.878293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtjr\" (UniqueName: \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\") pod \"payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.888669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.888264    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:39.888669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.888305    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276bl\" (UniqueName: \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:39.910681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.910659    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:39.990778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.990192    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:39.990778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.990268    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276bl\" (UniqueName: \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:39.992417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:39.992333    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:40.048483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:40.048454    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-276bl\" (UniqueName: \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\") pod \"payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:40.285021 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:40.284992    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93e43ca_7512_480e_9126_fed363b62908.slice/crio-fc4398a9eb7244fc09a589763caa706f8799153d39db57c769b182ab9848af44 WatchSource:0}: Error finding container fc4398a9eb7244fc09a589763caa706f8799153d39db57c769b182ab9848af44: Status 404 returned error can't find the container with id fc4398a9eb7244fc09a589763caa706f8799153d39db57c769b182ab9848af44
Nov 03 21:31:40.286445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:40.286415    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 21:31:40.331995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:40.331974    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:40.711942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:40.711914    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 21:31:40.712308 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:40.712138    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27255467_617e_4861_a359_b750691c1f84.slice/crio-e95b0db8fd404b7299634f9cd00a4171400e7c8caa815a0b150eceeb37d9193d WatchSource:0}: Error finding container e95b0db8fd404b7299634f9cd00a4171400e7c8caa815a0b150eceeb37d9193d: Status 404 returned error can't find the container with id e95b0db8fd404b7299634f9cd00a4171400e7c8caa815a0b150eceeb37d9193d
Nov 03 21:31:41.053979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.053885    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" event={"ID":"27255467-617e-4861-a359-b750691c1f84","Type":"ContainerStarted","Data":"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"}
Nov 03 21:31:41.053979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.053952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" event={"ID":"27255467-617e-4861-a359-b750691c1f84","Type":"ContainerStarted","Data":"e95b0db8fd404b7299634f9cd00a4171400e7c8caa815a0b150eceeb37d9193d"}
Nov 03 21:31:41.054191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.054134    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:41.055678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.055627    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" event={"ID":"a93e43ca-7512-480e-9126-fed363b62908","Type":"ContainerStarted","Data":"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"}
Nov 03 21:31:41.055794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.055697    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" event={"ID":"a93e43ca-7512-480e-9126-fed363b62908","Type":"ContainerStarted","Data":"fc4398a9eb7244fc09a589763caa706f8799153d39db57c769b182ab9848af44"}
Nov 03 21:31:41.055794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.055726    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:41.167295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.167234    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" podStartSLOduration=2.167216587 podStartE2EDuration="2.167216587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:41.16302429 +0000 UTC m=+2602.794337768" watchObservedRunningTime="2026-11-03 21:31:41.167216587 +0000 UTC m=+2602.798530069"
Nov 03 21:31:41.275014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.274955    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" podStartSLOduration=2.274936188 podStartE2EDuration="2.274936188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:41.272708085 +0000 UTC m=+2602.904021564" watchObservedRunningTime="2026-11-03 21:31:41.274936188 +0000 UTC m=+2602.906249666"
Nov 03 21:31:41.296823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.296789    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 21:31:41.297045 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:41.297021    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy" containerID="cri-o://0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6" gracePeriod=30
Nov 03 21:31:46.562804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.562771    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 21:31:46.570294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570266    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570319    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570353    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570393    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570434    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570615    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570656    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:46.570809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570677    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570716    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.570809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570750    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\") pod \"76c10d5f-84ff-44c1-9364-372f9358ff91\" (UID: \"76c10d5f-84ff-44c1-9364-372f9358ff91\") "
Nov 03 21:31:46.571040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.570934    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:31:46.571141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571124    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:31:46.571355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571333    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:31:46.571423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571375    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c10d5f-84ff-44c1-9364-372f9358ff91-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.571423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571396    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.571423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571410    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.571666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.571505    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:31:46.573554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.573528    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:46.573726 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.573605    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:31:46.573999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.573984    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:31:46.574083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.574068    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm" pod "76c10d5f-84ff-44c1-9364-372f9358ff91" (UID: "76c10d5f-84ff-44c1-9364-372f9358ff91"). InnerVolumeSpecName "kube-api-access-vtztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:46.672832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672807    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.672832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672830    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.672979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672840    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtztm\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-kube-api-access-vtztm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.672979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672850    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.672979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672860    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c10d5f-84ff-44c1-9364-372f9358ff91-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:46.672979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:46.672873    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c10d5f-84ff-44c1-9364-372f9358ff91-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:47.111497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.111459    5533 generic.go:356] "Generic (PLEG): container finished" podID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerID="0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6" exitCode=0
Nov 03 21:31:47.111755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.111535    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"
Nov 03 21:31:47.111755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.111554    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" event={"ID":"76c10d5f-84ff-44c1-9364-372f9358ff91","Type":"ContainerDied","Data":"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"}
Nov 03 21:31:47.111755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.111674    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx" event={"ID":"76c10d5f-84ff-44c1-9364-372f9358ff91","Type":"ContainerDied","Data":"b25da6bf38e920b41347f89ab82314c0f73317bc2053ab9e3866d4cb36e5b692"}
Nov 03 21:31:47.111755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.111708    5533 scope.go:122] "RemoveContainer" containerID="0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"
Nov 03 21:31:47.126914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.126894    5533 scope.go:122] "RemoveContainer" containerID="0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"
Nov 03 21:31:47.127200 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:31:47.127179    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6\": container with ID starting with 0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6 not found: ID does not exist" containerID="0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"
Nov 03 21:31:47.127266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.127208    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6"} err="failed to get container status \"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6\": rpc error: code = NotFound desc = could not find container \"0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6\": container with ID starting with 0f1b6f65c7f2b56a92e09f18569ea5686448d65c731adc2ddefba23f55ddc8d6 not found: ID does not exist"
Nov 03 21:31:47.271504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.271470    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 21:31:47.275888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:47.275864    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-f7mxx"]
Nov 03 21:31:48.047777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:48.047746    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"]
Nov 03 21:31:48.093934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:48.093909    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86d35cae-xrlx7"]
Nov 03 21:31:48.496249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:48.496212    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" path="/var/lib/kubelet/pods/76c10d5f-84ff-44c1-9364-372f9358ff91/volumes"
Nov 03 21:31:48.496720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:48.496704    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1edc2c9-4bee-4033-b865-58cad1e6613d" path="/var/lib/kubelet/pods/e1edc2c9-4bee-4033-b865-58cad1e6613d/volumes"
Nov 03 21:31:49.261630 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:31:49.261596    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:31:49.940268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.940235    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 21:31:49.940865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.940849    5533 cpu_manager.go:414] "containerMap: removing container" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy"
Nov 03 21:31:49.940919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.940868    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy"
Nov 03 21:31:49.941010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.940971    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c10d5f-84ff-44c1-9364-372f9358ff91" containerName="istio-proxy"
Nov 03 21:31:49.946570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.946546    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:49.999067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:49.999034    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 21:31:50.015338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.015304    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbss\" (UniqueName: \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.015484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.015366    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.040776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.040750    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a6219b-dockercfg-z9wvf\""
Nov 03 21:31:50.040892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.040779    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a6219b\""
Nov 03 21:31:50.097657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.097076    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 21:31:50.102661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.101967    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.137657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.126799    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbss\" (UniqueName: \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.137657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.126887    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.137657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.127980    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.162043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.158554    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 21:31:50.217201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.211422    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbss\" (UniqueName: \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\") pod \"payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.229234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.229192    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbf9q\" (UniqueName: \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.229435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.229411    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.263451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.263420    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:50.343575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.342257    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.343575 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.342392    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbf9q\" (UniqueName: \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.344016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.343958    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.400731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.400694    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbf9q\" (UniqueName: \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\") pod \"payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.426548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.426521    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:50.661516 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:50.661486    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda9d121_a1e5_4398_bd9d_d0b12700dac6.slice/crio-3f7050df661d2da52b2bf038921cbba85252f2e337232966296a4dc0effabc19 WatchSource:0}: Error finding container 3f7050df661d2da52b2bf038921cbba85252f2e337232966296a4dc0effabc19: Status 404 returned error can't find the container with id 3f7050df661d2da52b2bf038921cbba85252f2e337232966296a4dc0effabc19
Nov 03 21:31:50.661985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.661958    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 21:31:50.796527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:50.796498    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 21:31:50.796663 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:50.796627    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf165d751_3520_4a6f_913a_107b290ed114.slice/crio-4136a26ddf71e51e9baf6967ee6084970af202f64588b670af3fc601d22d078e WatchSource:0}: Error finding container 4136a26ddf71e51e9baf6967ee6084970af202f64588b670af3fc601d22d078e: Status 404 returned error can't find the container with id 4136a26ddf71e51e9baf6967ee6084970af202f64588b670af3fc601d22d078e
Nov 03 21:31:51.170568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.170534    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" event={"ID":"f165d751-3520-4a6f-913a-107b290ed114","Type":"ContainerStarted","Data":"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"}
Nov 03 21:31:51.170782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.170603    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:31:51.170782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.170627    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" event={"ID":"f165d751-3520-4a6f-913a-107b290ed114","Type":"ContainerStarted","Data":"4136a26ddf71e51e9baf6967ee6084970af202f64588b670af3fc601d22d078e"}
Nov 03 21:31:51.172962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.172926    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" event={"ID":"cda9d121-a1e5-4398-bd9d-d0b12700dac6","Type":"ContainerStarted","Data":"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"}
Nov 03 21:31:51.173099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.172991    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:31:51.173099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.173017    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" event={"ID":"cda9d121-a1e5-4398-bd9d-d0b12700dac6","Type":"ContainerStarted","Data":"3f7050df661d2da52b2bf038921cbba85252f2e337232966296a4dc0effabc19"}
Nov 03 21:31:51.280720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.280674    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" podStartSLOduration=1.280661478 podStartE2EDuration="1.280661478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:51.278975776 +0000 UTC m=+2612.910289253" watchObservedRunningTime="2026-11-03 21:31:51.280661478 +0000 UTC m=+2612.911974946"
Nov 03 21:31:51.391777 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:51.391707    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" podStartSLOduration=2.391690393 podStartE2EDuration="2.391690393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:51.387499283 +0000 UTC m=+2613.018812761" watchObservedRunningTime="2026-11-03 21:31:51.391690393 +0000 UTC m=+2613.023003871"
Nov 03 21:31:52.067554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:52.067524    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 21:31:52.067781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:52.067661    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 21:31:55.131162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.129200    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 21:31:55.141656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.138797    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.197663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.188426    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 21:31:55.228140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.227831    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-4d1799-dockercfg-wl9tk\""
Nov 03 21:31:55.238649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.228679    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-4d1799\""
Nov 03 21:31:55.239219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.239183    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zkn\" (UniqueName: \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.239408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.239387    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.303851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.303817    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 21:31:55.307934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.307917    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.340426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.340397    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zkn\" (UniqueName: \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.340548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.340537    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.341180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.341158    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.363689 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.361544    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 21:31:55.401089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.401026    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zkn\" (UniqueName: \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\") pod \"payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.441767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.441742    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.441914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.441783    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccfh\" (UniqueName: \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.475687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.475662    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:55.548482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.543945    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.548482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.544013    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccfh\" (UniqueName: \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.548482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.545630    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.603992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.603959    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccfh\" (UniqueName: \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\") pod \"payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.621807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.621780    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:55.875825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:55.875765    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 21:31:56.001960 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:31:56.001934    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3558e41_3e6a_47c8_84a7_1b068ca3285d.slice/crio-43668c99791701ebaa9d6cd82834e2d4e45d7cc09de508a9ee6a7f29ac73ba98 WatchSource:0}: Error finding container 43668c99791701ebaa9d6cd82834e2d4e45d7cc09de508a9ee6a7f29ac73ba98: Status 404 returned error can't find the container with id 43668c99791701ebaa9d6cd82834e2d4e45d7cc09de508a9ee6a7f29ac73ba98
Nov 03 21:31:56.002904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.002882    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 21:31:56.275954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.275881    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" event={"ID":"5b006e79-d578-41a5-92c1-de0e5cf3551e","Type":"ContainerStarted","Data":"dd23d53e62a4828bfe74569a177c7a64202fae5f523d295c9bb462ecf9ac73b6"}
Nov 03 21:31:56.275954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.275948    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:31:56.276378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.275974    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" event={"ID":"5b006e79-d578-41a5-92c1-de0e5cf3551e","Type":"ContainerStarted","Data":"559fbfaf738421efccf3fea475fc0bc84e46c996596cbc8e7dfae26f553679e7"}
Nov 03 21:31:56.277794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.277761    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" event={"ID":"f3558e41-3e6a-47c8-84a7-1b068ca3285d","Type":"ContainerStarted","Data":"c8f8fcf2cb55db9e3df150e92b00ee1ba8d676562c8022f465934a10ed7982c8"}
Nov 03 21:31:56.277928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.277817    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" event={"ID":"f3558e41-3e6a-47c8-84a7-1b068ca3285d","Type":"ContainerStarted","Data":"43668c99791701ebaa9d6cd82834e2d4e45d7cc09de508a9ee6a7f29ac73ba98"}
Nov 03 21:31:56.277928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.277863    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:31:56.444170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.444117    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" podStartSLOduration=1.444102169 podStartE2EDuration="1.444102169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:56.442300818 +0000 UTC m=+2618.073614295" watchObservedRunningTime="2026-11-03 21:31:56.444102169 +0000 UTC m=+2618.075415645"
Nov 03 21:31:56.552276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:56.552189    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" podStartSLOduration=1.552175107 podStartE2EDuration="1.552175107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:56.551232435 +0000 UTC m=+2618.182545937" watchObservedRunningTime="2026-11-03 21:31:56.552175107 +0000 UTC m=+2618.183488584"
Nov 03 21:31:59.047469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:59.047436    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"]
Nov 03 21:31:59.095978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:31:59.095947    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97d315fd-ktxrb"]
Nov 03 21:32:00.261255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.261214    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 21:32:00.266684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.266663    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.327397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.317558    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f0dc66-openshift-default-dockercfg-9d2q2\""
Nov 03 21:32:00.327682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.321369    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.327882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.327857    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.328055 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.328037    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.328277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.328260    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.328534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.328503    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.328776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.328755    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.328964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.328947    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.329264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.329235    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ggf\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.329444 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.329428    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.399725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.399691    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 21:32:00.432511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432468    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ggf\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432528    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432598    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432623    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432672    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432702    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432764    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432805    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.432896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.432837    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.433184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.433167    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.433965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.433933    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.434227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.434205    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.434487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.434468    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.434662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.434620    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.436295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.436275    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.436431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.436409    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.492503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.492392    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.493259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.493230    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ggf\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\") pod \"e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") " pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:00.502595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.502387    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d054fba9-a7be-429c-9c97-bcb4327f7494" path="/var/lib/kubelet/pods/d054fba9-a7be-429c-9c97-bcb4327f7494/volumes"
Nov 03 21:32:00.585133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:00.585044    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:01.017023 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:32:01.016993    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2502850_affd_4c72_8e42_adce28381d06.slice/crio-0d3414dd9d160ce64a73ba14dd658537a972bf522c72ae8abe34c42e03633490 WatchSource:0}: Error finding container 0d3414dd9d160ce64a73ba14dd658537a972bf522c72ae8abe34c42e03633490: Status 404 returned error can't find the container with id 0d3414dd9d160ce64a73ba14dd658537a972bf522c72ae8abe34c42e03633490
Nov 03 21:32:01.017925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.017895    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 21:32:01.019995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.019963    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:01.020073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.020024    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:01.020073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.020053    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:01.393890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.393809    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerStarted","Data":"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"}
Nov 03 21:32:01.393890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.393863    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerStarted","Data":"0d3414dd9d160ce64a73ba14dd658537a972bf522c72ae8abe34c42e03633490"}
Nov 03 21:32:01.509756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.509617    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podStartSLOduration=1.509599605 podStartE2EDuration="1.509599605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:01.507905086 +0000 UTC m=+2623.139218577" watchObservedRunningTime="2026-11-03 21:32:01.509599605 +0000 UTC m=+2623.140913082"
Nov 03 21:32:01.585931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:01.585903    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:02.191317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:02.191274    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 21:32:02.191495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:02.191366    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 21:32:02.591148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:02.591067    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:03.409268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:03.409229    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:03.410231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:03.410209    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:32:07.288481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:07.288450    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 21:32:07.288943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:07.288521    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 21:32:13.100569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.100531    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:32:13.106245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.106220    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.203982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.203940    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\""
Nov 03 21:32:13.215917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.215884    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:32:13.226169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226136    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226173    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226214    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226320    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226435    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226543    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2js\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226667    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226724    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.226868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.226850    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328176    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328236    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328276    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328292    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328315    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328333    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328399    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328474    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2js\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328552    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328772    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328808    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328877    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.328984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.328889    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.329131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.329111    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.330755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.330733    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.330893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.330875    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.386418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.386354    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2js\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.386418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.386365    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.424215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.423822    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:13.878311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.878273    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:32:13.878812 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:32:13.878769    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8463a1_788a_47c6_97ed_050a9a476698.slice/crio-8057ede3c185d80a4210845139052c59b9acece7a253a799ddc7954aa5f0bcbe WatchSource:0}: Error finding container 8057ede3c185d80a4210845139052c59b9acece7a253a799ddc7954aa5f0bcbe: Status 404 returned error can't find the container with id 8057ede3c185d80a4210845139052c59b9acece7a253a799ddc7954aa5f0bcbe
Nov 03 21:32:13.881465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.881431    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:13.881562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.881499    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:13.881562 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:13.881525    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:14.571925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:14.571877    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" event={"ID":"4f8463a1-788a-47c6-97ed-050a9a476698","Type":"ContainerStarted","Data":"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"}
Nov 03 21:32:14.572402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:14.571955    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" event={"ID":"4f8463a1-788a-47c6-97ed-050a9a476698","Type":"ContainerStarted","Data":"8057ede3c185d80a4210845139052c59b9acece7a253a799ddc7954aa5f0bcbe"}
Nov 03 21:32:14.732708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:14.728594    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podStartSLOduration=1.728571771 podStartE2EDuration="1.728571771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:14.715995231 +0000 UTC m=+2636.347308723" watchObservedRunningTime="2026-11-03 21:32:14.728571771 +0000 UTC m=+2636.359885252"
Nov 03 21:32:15.048212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:15.048171    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"]
Nov 03 21:32:15.080660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:15.080194    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-013dba96-dzhlh"]
Nov 03 21:32:15.424612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:15.424583    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:15.425874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:15.425848    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:15.426012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:15.425893    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:16.425110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:16.425072    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:16.425519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:16.425134    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:16.497969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:16.497930    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca7ae52-f4b7-4c32-9213-09eae40319b6" path="/var/lib/kubelet/pods/1ca7ae52-f4b7-4c32-9213-09eae40319b6/volumes"
Nov 03 21:32:17.425075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:17.425041    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:17.425257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:17.425108    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:18.425276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:18.425243    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:18.425687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:18.425306    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:19.284402 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:32:19.284365    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:32:19.424682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:19.424629    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:19.424861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:19.424711    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:20.048752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.048719    5533 scope.go:122] "RemoveContainer" containerID="55bf61a1d8ba79c12ffbaddb2b47bbaedd3050ad10f7bebb521b4762c7c8c720"
Nov 03 21:32:20.068266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.068241    5533 scope.go:122] "RemoveContainer" containerID="77bd309f4f2a5380e1603da54b24bdfdda9c84216b63462795364f450a89d1ab"
Nov 03 21:32:20.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.079301    5533 scope.go:122] "RemoveContainer" containerID="40101cf4c486369ba2afdb6addfded141278bfbcbdbb990176bf01f3c29cf99c"
Nov 03 21:32:20.089142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.089124    5533 scope.go:122] "RemoveContainer" containerID="9483d52ab97e1d32a32fe97493983d1b28e17ec4142c661efed7fab050115243"
Nov 03 21:32:20.425077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.425048    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 21:32:20.425226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:20.425099    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 21:32:21.429227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:21.429196    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:21.429715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:21.429684    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:21.430416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:21.430395    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:32:32.094675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:32.090893    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"]
Nov 03 21:32:32.103131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:32.103096    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5401116-x6c8c"]
Nov 03 21:32:32.505664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:32.505617    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d341475c-7919-4e67-b604-1b0178f46f66" path="/var/lib/kubelet/pods/d341475c-7919-4e67-b604-1b0178f46f66/volumes"
Nov 03 21:32:36.717124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:36.717086    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:32:42.048353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:42.048317    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"]
Nov 03 21:32:42.096919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:42.096885    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1937ac6-clmnd"]
Nov 03 21:32:42.495601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:42.495559    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea2860b-4206-43b2-a4cc-935b28da24ec" path="/var/lib/kubelet/pods/fea2860b-4206-43b2-a4cc-935b28da24ec/volumes"
Nov 03 21:32:45.979797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:45.979757    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:45.980201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:45.979826    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:47.919676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.919572    5533 generic.go:356] "Generic (PLEG): container finished" podID="0fe62722-5163-49cb-9d8d-2704bace267e" containerID="7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185" exitCode=0
Nov 03 21:32:47.919676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.919656    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerDied","Data":"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"}
Nov 03 21:32:47.920093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.920021    5533 scope.go:122] "RemoveContainer" containerID="7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"
Nov 03 21:32:47.921542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.921516    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:47.921607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.921578    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:47.921670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:47.921611    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:32:48.930560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:48.930525    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerStarted","Data":"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"}
Nov 03 21:32:49.316108 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:32:49.316019    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:32:49.623854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:49.623773    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:32:49.624033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:49.623921    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 21:32:49.624033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:49.623976    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 21:32:50.624656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:50.623966    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 21:32:50.624656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:50.624066    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 21:32:51.624322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:51.624287    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 21:32:51.624510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:51.624350    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 21:32:52.049669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:52.049625    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"]
Nov 03 21:32:52.096379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:52.096341    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f4d17-bm2tk"]
Nov 03 21:32:52.495271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:52.495246    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33aa2ed-c57c-447d-a89a-6e2208b6082a" path="/var/lib/kubelet/pods/a33aa2ed-c57c-447d-a89a-6e2208b6082a/volumes"
Nov 03 21:32:52.624233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:52.624203    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 21:32:52.624433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:52.624281    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 21:32:53.623767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:53.623733    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Nov 03 21:32:53.624226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:53.623808    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Nov 03 21:32:54.628106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:54.628076    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:32:54.628529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:54.628469    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:32:54.629163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:54.629141    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 21:32:57.030687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:57.030655    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:32:57.140838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:32:57.140754    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:33:02.031576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.031528    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" podUID="7989c489-429e-4fd5-b201-7184b8d23c84" containerName="payload-processing" containerID="cri-o://66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100" gracePeriod=25
Nov 03 21:33:02.141524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.141488    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" podUID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" containerName="payload-pre-processing" containerID="cri-o://181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09" gracePeriod=25
Nov 03 21:33:02.310486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.310458    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:33:02.418799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.418771    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:33:02.458252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.458224    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\" (UniqueName: \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\") pod \"7989c489-429e-4fd5-b201-7184b8d23c84\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") "
Nov 03 21:33:02.458424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.458400    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\") pod \"7989c489-429e-4fd5-b201-7184b8d23c84\" (UID: \"7989c489-429e-4fd5-b201-7184b8d23c84\") "
Nov 03 21:33:02.458693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.458676    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume" pod "7989c489-429e-4fd5-b201-7184b8d23c84" (UID: "7989c489-429e-4fd5-b201-7184b8d23c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:02.458943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.458923    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7989c489-429e-4fd5-b201-7184b8d23c84-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:02.460490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.460464    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn" pod "7989c489-429e-4fd5-b201-7184b8d23c84" (UID: "7989c489-429e-4fd5-b201-7184b8d23c84"). InnerVolumeSpecName "kube-api-access-flvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:02.560104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.560031    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\") pod \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") "
Nov 03 21:33:02.560104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.560101    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\" (UniqueName: \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\") pod \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\" (UID: \"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78\") "
Nov 03 21:33:02.560760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.560454    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume" pod "0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" (UID: "0e83dd9e-0cbf-4bff-bd69-db6e1f347e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:33:02.560760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.560629    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flvpn\" (UniqueName: \"kubernetes.io/projected/7989c489-429e-4fd5-b201-7184b8d23c84-kube-api-access-flvpn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:02.560760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.560669    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:02.563019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.562998    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd" pod "0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" (UID: "0e83dd9e-0cbf-4bff-bd69-db6e1f347e78"). InnerVolumeSpecName "kube-api-access-974nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:02.661595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:02.661563    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974nd\" (UniqueName: \"kubernetes.io/projected/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78-kube-api-access-974nd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:03.079267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.079229    5533 generic.go:356] "Generic (PLEG): container finished" podID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" containerID="181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09" exitCode=0
Nov 03 21:33:03.079714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.079271    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" event={"ID":"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78","Type":"ContainerDied","Data":"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"}
Nov 03 21:33:03.079714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.079300    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"
Nov 03 21:33:03.079714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.079370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f" event={"ID":"0e83dd9e-0cbf-4bff-bd69-db6e1f347e78","Type":"ContainerDied","Data":"04ec21fc9d25059f35db2948f4a3e09e0704af75aff0b2dc77059d6915ff1edb"}
Nov 03 21:33:03.079714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.079407    5533 scope.go:122] "RemoveContainer" containerID="181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"
Nov 03 21:33:03.081709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.081687    5533 generic.go:356] "Generic (PLEG): container finished" podID="7989c489-429e-4fd5-b201-7184b8d23c84" containerID="66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100" exitCode=0
Nov 03 21:33:03.081841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.081727    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" event={"ID":"7989c489-429e-4fd5-b201-7184b8d23c84","Type":"ContainerDied","Data":"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"}
Nov 03 21:33:03.081841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.081766    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm" event={"ID":"7989c489-429e-4fd5-b201-7184b8d23c84","Type":"ContainerDied","Data":"f2bc3efc9f0b29c22660f8d6bcce6b0614fc25157759a0f12d63760729430bca"}
Nov 03 21:33:03.081841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.081768    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"
Nov 03 21:33:03.091496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.091479    5533 scope.go:122] "RemoveContainer" containerID="181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"
Nov 03 21:33:03.091813 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:33:03.091795    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09\": container with ID starting with 181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09 not found: ID does not exist" containerID="181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"
Nov 03 21:33:03.091915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.091825    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09"} err="failed to get container status \"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09\": rpc error: code = NotFound desc = could not find container \"181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09\": container with ID starting with 181f3720545be66539ee32acfc8509e2e42153adf26fea61d4e0eec1594cfc09 not found: ID does not exist"
Nov 03 21:33:03.091915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.091850    5533 scope.go:122] "RemoveContainer" containerID="66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"
Nov 03 21:33:03.107731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.107713    5533 scope.go:122] "RemoveContainer" containerID="66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"
Nov 03 21:33:03.108013 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:33:03.107991    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100\": container with ID starting with 66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100 not found: ID does not exist" containerID="66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"
Nov 03 21:33:03.108091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.108020    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100"} err="failed to get container status \"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100\": rpc error: code = NotFound desc = could not find container \"66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100\": container with ID starting with 66a3c15948fa8d26e7c955df4838de32dad789f084e073f3a727a0c6a096e100 not found: ID does not exist"
Nov 03 21:33:03.236148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.236112    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:33:03.283584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.283540    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc4de5-58778c46c4-fdfzm"]
Nov 03 21:33:03.385804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.385722    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:33:03.389694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:03.389669    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cc4de5-856fd86cbc-4fl6f"]
Nov 03 21:33:04.495666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:04.495610    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" path="/var/lib/kubelet/pods/0e83dd9e-0cbf-4bff-bd69-db6e1f347e78/volumes"
Nov 03 21:33:04.496070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:04.496054    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7989c489-429e-4fd5-b201-7184b8d23c84" path="/var/lib/kubelet/pods/7989c489-429e-4fd5-b201-7184b8d23c84/volumes"
Nov 03 21:33:07.052655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:07.051729    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"]
Nov 03 21:33:07.094976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:07.094933    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15a972e4-jb5tq"]
Nov 03 21:33:08.495039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:08.494995    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c4d09c-08ba-4bb2-ac78-a3bed141bbd8" path="/var/lib/kubelet/pods/06c4d09c-08ba-4bb2-ac78-a3bed141bbd8/volumes"
Nov 03 21:33:14.047624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:14.047587    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"]
Nov 03 21:33:14.093754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:14.093726    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e75a582-mbq2l"]
Nov 03 21:33:14.494106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:14.494076    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1035f87b-6c1d-4203-b0be-1d8800c7eb3a" path="/var/lib/kubelet/pods/1035f87b-6c1d-4203-b0be-1d8800c7eb3a/volumes"
Nov 03 21:33:19.007593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:19.007563    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:33:19.019424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:19.019404    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:33:19.030151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:19.030131    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:33:19.041011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:19.040996    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:33:19.261456 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:33:19.261381    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:33:20.162351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.162302    5533 scope.go:122] "RemoveContainer" containerID="e56fd9c6261b9fca13cab8833e388108cd18f5be6dfb4d24d1ef176eb2bfbdef"
Nov 03 21:33:20.189940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.189914    5533 scope.go:122] "RemoveContainer" containerID="dbdcd468d7a4041085799fce3e0ada84949d37f353f995a0141dc8643398dcac"
Nov 03 21:33:20.207074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.206953    5533 scope.go:122] "RemoveContainer" containerID="af820c1dbf32f20da95c7a73555f7ae3eb387f192f4c3dd81bcf4565b1204779"
Nov 03 21:33:20.221839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.221804    5533 scope.go:122] "RemoveContainer" containerID="706935f9c1a523ce6e93c0b5e21730b8fc91801c3e86502d0a888242e49ec8b3"
Nov 03 21:33:20.233361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.233344    5533 scope.go:122] "RemoveContainer" containerID="b403e193fdbb3f7a236f681f41c702f2f8d79977a4a50ab728424ac8f529bdd1"
Nov 03 21:33:20.244464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:20.244443    5533 scope.go:122] "RemoveContainer" containerID="8fb3f6421cd525cc5fb425a393d364e18f9e4598eb71da05396a8229e828a2b2"
Nov 03 21:33:25.759657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:25.759603    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:33:28.678383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.678338    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689586    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7989c489-429e-4fd5-b201-7184b8d23c84" containerName="payload-processing"
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689611    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7989c489-429e-4fd5-b201-7184b8d23c84" containerName="payload-processing"
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689682    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" containerName="payload-pre-processing"
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689691    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" containerName="payload-pre-processing"
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689803    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e83dd9e-0cbf-4bff-bd69-db6e1f347e78" containerName="payload-pre-processing"
Nov 03 21:33:28.692696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.689819    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7989c489-429e-4fd5-b201-7184b8d23c84" containerName="payload-processing"
Nov 03 21:33:28.696188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.696159    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:28.736112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.736051    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:33:28.781956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.781924    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b22936\""
Nov 03 21:33:28.814647 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.814586    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b22936-dockercfg-kc4nq\""
Nov 03 21:33:28.906655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.902271    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:28.906655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:28.902338    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqhr\" (UniqueName: \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.003561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.003458    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.003561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.003513    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqhr\" (UniqueName: \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.004503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.004477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.046954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.046927    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"]
Nov 03 21:33:29.079464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.079437    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35a966c-pln5t"]
Nov 03 21:33:29.079594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.079516    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqhr\" (UniqueName: \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\") pod \"payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.313076 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.312975    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:29.708175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:29.708144    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:33:29.709085 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:33:29.709061    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1097f4_d542_409f_b39f_3acc2c40c690.slice/crio-25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf WatchSource:0}: Error finding container 25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf: Status 404 returned error can't find the container with id 25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf
Nov 03 21:33:30.318695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:30.317622    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" event={"ID":"2b1097f4-d542-409f-b39f-3acc2c40c690","Type":"ContainerStarted","Data":"5798b3c3613cc2ee524babff35c6cd4639ceda3e98fa73f1189432cdf2994fbe"}
Nov 03 21:33:30.318695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:30.317695    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" event={"ID":"2b1097f4-d542-409f-b39f-3acc2c40c690","Type":"ContainerStarted","Data":"25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf"}
Nov 03 21:33:30.318695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:30.317849    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:30.434698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:30.434649    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" podStartSLOduration=2.434620686 podStartE2EDuration="2.434620686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:30.43182626 +0000 UTC m=+2712.063139736" watchObservedRunningTime="2026-11-03 21:33:30.434620686 +0000 UTC m=+2712.065934162"
Nov 03 21:33:30.496401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:30.496364    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6257d3b6-086d-45fa-af3b-bc9f45cd547e" path="/var/lib/kubelet/pods/6257d3b6-086d-45fa-af3b-bc9f45cd547e/volumes"
Nov 03 21:33:34.451827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:34.451787    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:33:34.452404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:34.451852    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:36.390919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.390883    5533 generic.go:356] "Generic (PLEG): container finished" podID="c2502850-affd-4c72-8e42-adce28381d06" containerID="8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708" exitCode=0
Nov 03 21:33:36.391486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.390955    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerDied","Data":"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"}
Nov 03 21:33:36.391486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.391416    5533 scope.go:122] "RemoveContainer" containerID="8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"
Nov 03 21:33:36.393425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.393383    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:33:36.393520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.393494    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:33:36.393574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:36.393549    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:33:37.403299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:37.403263    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerStarted","Data":"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"}
Nov 03 21:33:37.585819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:37.585777    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:33:37.586033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:37.585992    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:37.586105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:37.586053    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:38.587659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:38.587137    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:38.587659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:38.587195    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:39.585648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:39.585601    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:39.585841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:39.585674    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:40.585470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:40.585417    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:40.585887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:40.585490    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:41.328520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:41.328489    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:33:41.588168 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:41.588072    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:41.588867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:41.588733    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:42.587012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:42.586976    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Nov 03 21:33:42.587331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:42.587299    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Nov 03 21:33:43.589985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:43.589952    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:33:43.590495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:43.590479    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:33:43.591226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:43.591207    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 21:33:49.049501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:49.049465    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"]
Nov 03 21:33:49.096242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:49.096212    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd31cc2-6cmld"]
Nov 03 21:33:49.261534 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:33:49.261498    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:33:50.494577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:50.494543    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc01fee-cfab-4640-8c4d-617e6df1458b" path="/var/lib/kubelet/pods/5dc01fee-cfab-4640-8c4d-617e6df1458b/volumes"
Nov 03 21:33:53.044991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:53.044957    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"]
Nov 03 21:33:53.091577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:53.091545    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6590b997-lvwvc"]
Nov 03 21:33:54.495653 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:33:54.495603    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d123c24-13bf-480f-8121-2c618acc230e" path="/var/lib/kubelet/pods/9d123c24-13bf-480f-8121-2c618acc230e/volumes"
Nov 03 21:34:00.412157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:00.412121    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:34:05.048467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:05.048433    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"]
Nov 03 21:34:05.094046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:05.094014    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c4a63e9-vqzvk"]
Nov 03 21:34:06.495062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:06.495019    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bc6efd-fa21-4f18-af58-fbfff88f4acd" path="/var/lib/kubelet/pods/33bc6efd-fa21-4f18-af58-fbfff88f4acd/volumes"
Nov 03 21:34:08.589762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.589728    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 21:34:08.596232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.596210    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.653477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.653437    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 21:34:08.749658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.719160    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-714979-dockercfg-6bd6t\""
Nov 03 21:34:08.775662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.731041    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-714979\""
Nov 03 21:34:08.775662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.732567    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.775662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.774191    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2rf\" (UniqueName: \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.828658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.816975    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 21:34:08.828658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.822307    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:08.849360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.849282    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 21:34:08.880056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.880020    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2rf\" (UniqueName: \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.880191 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.880083    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:08.880344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.880316    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.880568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.880546    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8kv\" (UniqueName: \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:08.882032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.882008    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.938852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.938829    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2rf\" (UniqueName: \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\") pod \"payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:08.981750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.981725    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8kv\" (UniqueName: \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:08.981863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.981785    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:08.982704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:08.982672    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:09.040724 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.040694    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8kv\" (UniqueName: \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\") pod \"payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:09.046858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.046835    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"]
Nov 03 21:34:09.050291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.050270    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9f9212a-sthtv"]
Nov 03 21:34:09.189240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.189215    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:09.210180 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.209985    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:09.569716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.569675    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 21:34:09.571017 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:34:09.570986    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbc05dc_c298_4805_9710_1c770130d79d.slice/crio-e31f02719af6c1fad22f163c71e42b82193dab5d66c4eaac9ea8f2f8adcaa69f WatchSource:0}: Error finding container e31f02719af6c1fad22f163c71e42b82193dab5d66c4eaac9ea8f2f8adcaa69f: Status 404 returned error can't find the container with id e31f02719af6c1fad22f163c71e42b82193dab5d66c4eaac9ea8f2f8adcaa69f
Nov 03 21:34:09.613996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.613970    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 21:34:09.615320 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:34:09.615292    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac17cc0a_903d_47af_a5da_24f205a228e5.slice/crio-2671f3d55b20dd61b1d8cc486070e1a412623aadae023e9075325a52d6b63bc4 WatchSource:0}: Error finding container 2671f3d55b20dd61b1d8cc486070e1a412623aadae023e9075325a52d6b63bc4: Status 404 returned error can't find the container with id 2671f3d55b20dd61b1d8cc486070e1a412623aadae023e9075325a52d6b63bc4
Nov 03 21:34:09.659945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.659911    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" event={"ID":"dbbc05dc-c298-4805-9710-1c770130d79d","Type":"ContainerStarted","Data":"247e9b2871799f4d520838845828eebef1110f62c104e84e036fab5ee5eeba2d"}
Nov 03 21:34:09.660059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.659986    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:09.660059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.660010    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" event={"ID":"dbbc05dc-c298-4805-9710-1c770130d79d","Type":"ContainerStarted","Data":"e31f02719af6c1fad22f163c71e42b82193dab5d66c4eaac9ea8f2f8adcaa69f"}
Nov 03 21:34:09.661560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.661529    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" event={"ID":"ac17cc0a-903d-47af-a5da-24f205a228e5","Type":"ContainerStarted","Data":"2671f3d55b20dd61b1d8cc486070e1a412623aadae023e9075325a52d6b63bc4"}
Nov 03 21:34:09.771610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:09.771561    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" podStartSLOduration=1.771545006 podStartE2EDuration="1.771545006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:09.771004352 +0000 UTC m=+2751.402317829" watchObservedRunningTime="2026-11-03 21:34:09.771545006 +0000 UTC m=+2751.402858483"
Nov 03 21:34:10.496705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:10.496671    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7" path="/var/lib/kubelet/pods/8f9fd7e3-3c93-4d99-908e-84f25f3fa2d7/volumes"
Nov 03 21:34:10.670544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:10.670508    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" event={"ID":"ac17cc0a-903d-47af-a5da-24f205a228e5","Type":"ContainerStarted","Data":"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"}
Nov 03 21:34:10.671000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:10.670761    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:10.780247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:10.780148    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" podStartSLOduration=2.78013155 podStartE2EDuration="2.78013155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:10.779296573 +0000 UTC m=+2752.410610049" watchObservedRunningTime="2026-11-03 21:34:10.78013155 +0000 UTC m=+2752.411445030"
Nov 03 21:34:19.262027 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:34:19.261990    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:34:20.371892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.371854    5533 scope.go:122] "RemoveContainer" containerID="8a91c0dc96c6dc1fba1f37f2085c65c030bd99ee2ffa86303a6fddb62c1b0582"
Nov 03 21:34:20.384177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.384156    5533 scope.go:122] "RemoveContainer" containerID="6615dbd972dd042b92bfe10e573c9b9ab29205947162c1c0e0f9a8abb5038182"
Nov 03 21:34:20.394821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.394797    5533 scope.go:122] "RemoveContainer" containerID="21aae396a82634810f1d22d0ad37da0f4b142c4d8be2fa0d82beb7ecbf5571a4"
Nov 03 21:34:20.403528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.403504    5533 scope.go:122] "RemoveContainer" containerID="3b668384181eb84dd44d344def8f678e65d6db7c0f67ef3943301af7cfecdd05"
Nov 03 21:34:20.412614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.412565    5533 scope.go:122] "RemoveContainer" containerID="c838c70cd52c78642264b65d2e91a09364e3b8843f16510c581a83a19ffba583"
Nov 03 21:34:20.673335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:20.673311    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 21:34:21.684478 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:21.684450    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 21:34:26.046509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.046472    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"]
Nov 03 21:34:26.093690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.093659    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5a6a6-psc55"]
Nov 03 21:34:26.097119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.097087    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"]
Nov 03 21:34:26.100092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.100064    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e54e645-g84ll"]
Nov 03 21:34:26.495188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.495158    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb1aeee-de4b-425d-8f98-3c977246c615" path="/var/lib/kubelet/pods/5fb1aeee-de4b-425d-8f98-3c977246c615/volumes"
Nov 03 21:34:26.495719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:26.495701    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8582affc-2bd8-4d4f-9ef7-fc280faf6aba" path="/var/lib/kubelet/pods/8582affc-2bd8-4d4f-9ef7-fc280faf6aba/volumes"
Nov 03 21:34:37.051350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:37.051310    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"]
Nov 03 21:34:37.095631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:37.095602    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008dad2c-8zwgx"]
Nov 03 21:34:38.494275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:38.494241    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e" path="/var/lib/kubelet/pods/5a905ac7-2b01-4c1f-9232-d8bbdc93cb2e/volumes"
Nov 03 21:34:48.750354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:48.750319    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:34:49.262969 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:34:49.262936    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:34:54.057953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:54.057916    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"]
Nov 03 21:34:54.096847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:54.096815    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c08fe4-bvv6j"]
Nov 03 21:34:54.494544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:54.494512    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b774c1-1ed3-4a04-9861-244ba96d259c" path="/var/lib/kubelet/pods/d8b774c1-1ed3-4a04-9861-244ba96d259c/volumes"
Nov 03 21:34:58.546170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.546137    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:34:58.550579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.550559    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.597764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.597734    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:34:58.639922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.639896    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"openshift-service-ca.crt\""
Nov 03 21:34:58.639922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.639905    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"auth-test-model-c11538-kserve-self-signed-certs\""
Nov 03 21:34:58.640130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.639933    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"default-dockercfg-z6nqg\""
Nov 03 21:34:58.646738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.646711    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"kube-root-ca.crt\""
Nov 03 21:34:58.693610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693583    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.693738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693619    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.693738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693702    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.693738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693729    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.693887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693769    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlc9n\" (UniqueName: \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.693975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.693955    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.795609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795573    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.795831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795784    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.795831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795818    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.795960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795860    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.795960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795893    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.796071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.795960    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlc9n\" (UniqueName: \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.796071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.796009    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.796157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.796111    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.796288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.796238    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.798345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.798317    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.798815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.798798    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.854043 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.854012    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlc9n\" (UniqueName: \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\") pod \"auth-test-model-c11538-kserve-7c5677fc4f-dd2pk\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") " pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:58.865928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:58.865897    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:34:59.241821 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:34:59.241791    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ad35a0_76fa_4103_9118_f7acf9602c9f.slice/crio-d172a470f5b27bbf7b5f5500f76eb651b82073e91af908d6869ff40c12a99504 WatchSource:0}: Error finding container d172a470f5b27bbf7b5f5500f76eb651b82073e91af908d6869ff40c12a99504: Status 404 returned error can't find the container with id d172a470f5b27bbf7b5f5500f76eb651b82073e91af908d6869ff40c12a99504
Nov 03 21:34:59.242389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:34:59.242360    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:35:00.084430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:00.084384    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" event={"ID":"39ad35a0-76fa-4103-9118-f7acf9602c9f","Type":"ContainerStarted","Data":"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"}
Nov 03 21:35:00.084430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:00.084433    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" event={"ID":"39ad35a0-76fa-4103-9118-f7acf9602c9f","Type":"ContainerStarted","Data":"d172a470f5b27bbf7b5f5500f76eb651b82073e91af908d6869ff40c12a99504"}
Nov 03 21:35:00.200557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:00.200497    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" podStartSLOduration=2.20048009 podStartE2EDuration="2.20048009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:00.194666073 +0000 UTC m=+2801.825979550" watchObservedRunningTime="2026-11-03 21:35:00.20048009 +0000 UTC m=+2801.831793572"
Nov 03 21:35:00.866544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:00.866515    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:35:08.866652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:08.866608    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:35:08.899492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:08.899456    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:35:09.159740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:09.159664    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:35:19.262158 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:19.262120    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:35:20.461890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:20.461855    5533 scope.go:122] "RemoveContainer" containerID="76bb45a7ab8ed6b32742cb81c7e5e34d54ebdcb2dfebee7a604c68c36ff7af4e"
Nov 03 21:35:20.472371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:20.472346    5533 scope.go:122] "RemoveContainer" containerID="5c4711d32b7049d5536513361a4510d8444a0a84256c13055213c6108cc09383"
Nov 03 21:35:20.484441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:20.484419    5533 scope.go:122] "RemoveContainer" containerID="b29ef64a79b9ad68cc72558e3a0cd697c3e53758d5f2a7da70655ed3282818ed"
Nov 03 21:35:20.496683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:20.496661    5533 scope.go:122] "RemoveContainer" containerID="9a86522a3a3b9869c87e0d403c5a2cc5e4805950a96a314451a2575e9b2749c1"
Nov 03 21:35:20.505611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:20.505591    5533 scope.go:122] "RemoveContainer" containerID="9d6bd64807b5307c7caf3929737430ee065e78d7788a21ab2b925d00730c0c7e"
Nov 03 21:35:26.709876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.709836    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:35:26.710373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.710029    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy" containerID="cri-o://801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7" gracePeriod=30
Nov 03 21:35:26.756964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.756938    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:35:26.757181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.757158    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy" containerID="cri-o://264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152" gracePeriod=30
Nov 03 21:35:26.761665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.761625    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:35:26.761838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:26.761820    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" containerID="cri-o://403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b" gracePeriod=30
Nov 03 21:35:30.046123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:30.046086    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"]
Nov 03 21:35:30.072473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:30.072447    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54301df-6t729"]
Nov 03 21:35:30.494602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:30.494568    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe913e4-7347-4c38-b7b6-1920c6c0ac06" path="/var/lib/kubelet/pods/afe913e4-7347-4c38-b7b6-1920c6c0ac06/volumes"
Nov 03 21:35:31.293789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:31.293755    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:35:31.294157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:31.293809    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:35:32.050471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.050437    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:35:32.107515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107489    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107532    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107584    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107646 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107617    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107668    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107721    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107766    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.107989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107835    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107882    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107893    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.107989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107917    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\") pod \"83b3898b-a3a1-49a7-b753-3186a3b18414\" (UID: \"83b3898b-a3a1-49a7-b753-3186a3b18414\") "
Nov 03 21:35:32.107989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.107937    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.108275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108156    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.108275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108179    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:32.108573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108546    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.108718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108577    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.108718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108593    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.108718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108607    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.108718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.108622    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3898b-a3a1-49a7-b753-3186a3b18414-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.110301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.110275    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "kube-api-access-r9nkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.110599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.110570    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.110760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.110616    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:35:32.110760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.110654    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy" pod "83b3898b-a3a1-49a7-b753-3186a3b18414" (UID: "83b3898b-a3a1-49a7-b753-3186a3b18414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.114425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.114405    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:35:32.116866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.116849    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:35:32.209669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209624    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.209783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209681    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.209783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209713    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.209783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209747    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.209783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209772    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209820    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209853    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209887    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209891    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209918    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209992    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.209996    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210147    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210162    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210202    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210229    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210234    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\") pod \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\" (UID: \"8788126a-0f5a-48d5-bdbb-92ddf4074cb3\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210325    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210438    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210512    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210549    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210562    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210661    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210723    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210754    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210771    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.210917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.210808    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\") pod \"7f138ead-4210-4a04-bdad-e00a622afd4a\" (UID: \"7f138ead-4210-4a04-bdad-e00a622afd4a\") "
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211671    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211696    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211712    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211727    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211744    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nkp\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-kube-api-access-r9nkp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211760    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211774    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211810    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211826    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f138ead-4210-4a04-bdad-e00a622afd4a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211841    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211855    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211869    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.211880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211884    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.212535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.211902    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3898b-a3a1-49a7-b753-3186a3b18414-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.212535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.212492    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.212885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.212846    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:35:32.212991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.212953    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "kube-api-access-fvt24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.213142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.213118    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "kube-api-access-w7xw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.213232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.213205    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:32.213314 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.213251    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.213547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.213528    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy" pod "8788126a-0f5a-48d5-bdbb-92ddf4074cb3" (UID: "8788126a-0f5a-48d5-bdbb-92ddf4074cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:32.213825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.213800    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo" pod "7f138ead-4210-4a04-bdad-e00a622afd4a" (UID: "7f138ead-4210-4a04-bdad-e00a622afd4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312890    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312914    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-kube-api-access-w7xw5\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312925    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312936    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312945    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt24\" (UniqueName: \"kubernetes.io/projected/7f138ead-4210-4a04-bdad-e00a622afd4a-kube-api-access-fvt24\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.312956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312953    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.313456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312963    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f138ead-4210-4a04-bdad-e00a622afd4a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.313456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.312971    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8788126a-0f5a-48d5-bdbb-92ddf4074cb3-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:32.348668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.348606    5533 generic.go:356] "Generic (PLEG): container finished" podID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerID="264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152" exitCode=0
Nov 03 21:35:32.348837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.348666    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" event={"ID":"83b3898b-a3a1-49a7-b753-3186a3b18414","Type":"ContainerDied","Data":"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"}
Nov 03 21:35:32.348837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.348697    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"
Nov 03 21:35:32.348837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.348722    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn" event={"ID":"83b3898b-a3a1-49a7-b753-3186a3b18414","Type":"ContainerDied","Data":"58a26c8c7529b288e73a2c2718a5a016ab468fb52b64ea5ce12216e9b06dde80"}
Nov 03 21:35:32.348837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.348751    5533 scope.go:122] "RemoveContainer" containerID="264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"
Nov 03 21:35:32.350729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.350704    5533 generic.go:356] "Generic (PLEG): container finished" podID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerID="403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b" exitCode=0
Nov 03 21:35:32.350851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.350760    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"
Nov 03 21:35:32.350851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.350801    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" event={"ID":"7f138ead-4210-4a04-bdad-e00a622afd4a","Type":"ContainerDied","Data":"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"}
Nov 03 21:35:32.350992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.350856    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl" event={"ID":"7f138ead-4210-4a04-bdad-e00a622afd4a","Type":"ContainerDied","Data":"8914449a6860f0b80dd9e4afd8692d51604477f68049ff47c70796c34889b38d"}
Nov 03 21:35:32.352691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.352667    5533 generic.go:356] "Generic (PLEG): container finished" podID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerID="801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7" exitCode=0
Nov 03 21:35:32.352813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.352709    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"
Nov 03 21:35:32.352813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.352734    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" event={"ID":"8788126a-0f5a-48d5-bdbb-92ddf4074cb3","Type":"ContainerDied","Data":"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"}
Nov 03 21:35:32.352813 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.352774    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx" event={"ID":"8788126a-0f5a-48d5-bdbb-92ddf4074cb3","Type":"ContainerDied","Data":"9614c066a644080d824b68a5482bf21509e12813fdc922b675b52971b26ff575"}
Nov 03 21:35:32.377717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.377668    5533 scope.go:122] "RemoveContainer" containerID="264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"
Nov 03 21:35:32.378046 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:32.378019    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152\": container with ID starting with 264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152 not found: ID does not exist" containerID="264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"
Nov 03 21:35:32.378188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.378162    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152"} err="failed to get container status \"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152\": rpc error: code = NotFound desc = could not find container \"264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152\": container with ID starting with 264331c8203c1f00f9e7b4751924fc10cf182bd3225b4c28d24b727b3ea0d152 not found: ID does not exist"
Nov 03 21:35:32.378299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.378286    5533 scope.go:122] "RemoveContainer" containerID="403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"
Nov 03 21:35:32.389972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.389956    5533 scope.go:122] "RemoveContainer" containerID="403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"
Nov 03 21:35:32.390278 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:32.390253    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b\": container with ID starting with 403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b not found: ID does not exist" containerID="403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"
Nov 03 21:35:32.390353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.390288    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b"} err="failed to get container status \"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b\": rpc error: code = NotFound desc = could not find container \"403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b\": container with ID starting with 403bebdf7b3643424992bb26238ebb74fd08830d29f0dbc1f010577b02dd526b not found: ID does not exist"
Nov 03 21:35:32.390353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.390310    5533 scope.go:122] "RemoveContainer" containerID="801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"
Nov 03 21:35:32.401420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.401394    5533 scope.go:122] "RemoveContainer" containerID="801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"
Nov 03 21:35:32.401707 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:32.401688    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7\": container with ID starting with 801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7 not found: ID does not exist" containerID="801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"
Nov 03 21:35:32.401775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.401714    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7"} err="failed to get container status \"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7\": rpc error: code = NotFound desc = could not find container \"801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7\": container with ID starting with 801c6db47a0ef09f3469973056a6ab6778c3fcbf513bc6c47f2247d3df8621f7 not found: ID does not exist"
Nov 03 21:35:32.548864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.548830    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:35:32.552672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.552646    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-mdlmx"]
Nov 03 21:35:32.653854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.653776    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:35:32.661181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.661146    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-bjbcl"]
Nov 03 21:35:32.763268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.763242    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:35:32.768067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:32.768039    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-54mgn"]
Nov 03 21:35:34.494720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:34.494687    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" path="/var/lib/kubelet/pods/7f138ead-4210-4a04-bdad-e00a622afd4a/volumes"
Nov 03 21:35:34.495334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:34.495313    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" path="/var/lib/kubelet/pods/83b3898b-a3a1-49a7-b753-3186a3b18414/volumes"
Nov 03 21:35:34.495828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:34.495808    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" path="/var/lib/kubelet/pods/8788126a-0f5a-48d5-bdbb-92ddf4074cb3/volumes"
Nov 03 21:35:40.638328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:40.638291    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:35:45.109597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:45.109563    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:35:45.110079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:45.109789    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" containerID="cri-o://e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71" gracePeriod=30
Nov 03 21:35:45.158115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:45.158083    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:35:45.158332 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:45.158309    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy" containerID="cri-o://ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21" gracePeriod=30
Nov 03 21:35:49.263134 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:49.263093    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:35:50.445730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.445664    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:35:50.463498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.463471    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:35:50.511514 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.511489    5533 generic.go:356] "Generic (PLEG): container finished" podID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerID="e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71" exitCode=0
Nov 03 21:35:50.511657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.511535    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"
Nov 03 21:35:50.511657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.511566    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" event={"ID":"c661d9b0-04e9-4d64-953b-8e1be4b50929","Type":"ContainerDied","Data":"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"}
Nov 03 21:35:50.511657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.511614    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" event={"ID":"c661d9b0-04e9-4d64-953b-8e1be4b50929","Type":"ContainerDied","Data":"b1a6b50a9bfcdf7fe9d6a18694ba2db74caf2357f420cdabb3fb49c426c520f8"}
Nov 03 21:35:50.511657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.511656    5533 scope.go:122] "RemoveContainer" containerID="e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"
Nov 03 21:35:50.513118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.513096    5533 generic.go:356] "Generic (PLEG): container finished" podID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerID="ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21" exitCode=0
Nov 03 21:35:50.513204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.513134    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"
Nov 03 21:35:50.513204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.513169    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" event={"ID":"f20ceb59-6017-429b-a02b-63863f27d2d6","Type":"ContainerDied","Data":"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"}
Nov 03 21:35:50.513296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.513213    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv" event={"ID":"f20ceb59-6017-429b-a02b-63863f27d2d6","Type":"ContainerDied","Data":"2f6fefb172c7eeda33e61e1b8505b8f27a568dd5995039b3ab87bbf5b13272a6"}
Nov 03 21:35:50.521400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.521382    5533 scope.go:122] "RemoveContainer" containerID="e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"
Nov 03 21:35:50.521677 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:50.521649    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71\": container with ID starting with e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71 not found: ID does not exist" containerID="e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"
Nov 03 21:35:50.521789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.521685    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71"} err="failed to get container status \"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71\": rpc error: code = NotFound desc = could not find container \"e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71\": container with ID starting with e29938bb661b070c54218317a9c63cb1131997d5629ffbb3ce7814028d879e71 not found: ID does not exist"
Nov 03 21:35:50.521789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.521702    5533 scope.go:122] "RemoveContainer" containerID="ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"
Nov 03 21:35:50.530743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.530727    5533 scope.go:122] "RemoveContainer" containerID="ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"
Nov 03 21:35:50.531000 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:35:50.530981    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21\": container with ID starting with ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21 not found: ID does not exist" containerID="ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"
Nov 03 21:35:50.531071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.531006    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21"} err="failed to get container status \"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21\": rpc error: code = NotFound desc = could not find container \"ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21\": container with ID starting with ec11fd3215f3726ae937d47c0e8d6c6914d981ee45b41259de5a288de70e1b21 not found: ID does not exist"
Nov 03 21:35:50.532303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532285    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.532360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532316    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532345    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.532428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532375    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532397    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.532576 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532544    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.532661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532600    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.532729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532693    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.532775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532732    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532783    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532870 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532827    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532905    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.532926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532921    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532946    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.533056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.532974    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.533056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533022    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.533283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533063    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.533283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533169    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\") pod \"c661d9b0-04e9-4d64-953b-8e1be4b50929\" (UID: \"c661d9b0-04e9-4d64-953b-8e1be4b50929\") "
Nov 03 21:35:50.533283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533191    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533206    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.533283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533260    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\") pod \"f20ceb59-6017-429b-a02b-63863f27d2d6\" (UID: \"f20ceb59-6017-429b-a02b-63863f27d2d6\") "
Nov 03 21:35:50.533545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533391    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533467    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533488    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533535    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533732    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.533858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533821    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533952    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533978    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.533994    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534009    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534024    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534037    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534038    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534052    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534067    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534081    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20ceb59-6017-429b-a02b-63863f27d2d6-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.534897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534865    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.534997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.534917    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:50.535522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.535498    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "kube-api-access-8pd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:50.535612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.535534    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "kube-api-access-7989s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:50.535831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.535807    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo" pod "f20ceb59-6017-429b-a02b-63863f27d2d6" (UID: "f20ceb59-6017-429b-a02b-63863f27d2d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:35:50.535893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.535881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:35:50.536828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.536811    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:35:50.536997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.536982    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token" pod "c661d9b0-04e9-4d64-953b-8e1be4b50929" (UID: "c661d9b0-04e9-4d64-953b-8e1be4b50929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635088    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635112    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c661d9b0-04e9-4d64-953b-8e1be4b50929-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635123    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635132    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c661d9b0-04e9-4d64-953b-8e1be4b50929-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635142    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635155    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635164    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635173    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pd7m\" (UniqueName: \"kubernetes.io/projected/c661d9b0-04e9-4d64-953b-8e1be4b50929-kube-api-access-8pd7m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.635563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.635181    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7989s\" (UniqueName: \"kubernetes.io/projected/f20ceb59-6017-429b-a02b-63863f27d2d6-kube-api-access-7989s\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:35:50.931103 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.931069    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:35:50.934947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:50.934922    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w"]
Nov 03 21:35:51.039244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:51.039207    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:35:51.046356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:51.046330    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-9tbxv"]
Nov 03 21:35:51.439677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:51.439621    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:35:51.439849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:51.439702    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-q255w" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:52.494414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:52.494380    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" path="/var/lib/kubelet/pods/c661d9b0-04e9-4d64-953b-8e1be4b50929/volumes"
Nov 03 21:35:52.494858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:35:52.494843    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" path="/var/lib/kubelet/pods/f20ceb59-6017-429b-a02b-63863f27d2d6/volumes"
Nov 03 21:36:07.608456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.608420    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:36:07.608993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.608979    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy"
Nov 03 21:36:07.608993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.608993    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609004    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609009    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609023    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609028    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609045    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609050    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609061    5533 cpu_manager.go:414] "containerMap: removing container" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy"
Nov 03 21:36:07.609123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609066    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy"
Nov 03 21:36:07.609501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609167    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8788126a-0f5a-48d5-bdbb-92ddf4074cb3" containerName="istio-proxy"
Nov 03 21:36:07.609501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609175    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c661d9b0-04e9-4d64-953b-8e1be4b50929" containerName="istio-proxy"
Nov 03 21:36:07.609501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609183    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20ceb59-6017-429b-a02b-63863f27d2d6" containerName="istio-proxy"
Nov 03 21:36:07.609501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609192    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f138ead-4210-4a04-bdad-e00a622afd4a" containerName="istio-proxy"
Nov 03 21:36:07.609501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.609200    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b3898b-a3a1-49a7-b753-3186a3b18414" containerName="istio-proxy"
Nov 03 21:36:07.614126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.614103    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.710577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.710541    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"openshift-service-ca.crt\""
Nov 03 21:36:07.710795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.710772    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"auth-test-model-a6219b-kserve-self-signed-certs\""
Nov 03 21:36:07.710879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.710807    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"default-dockercfg-j4qpm\""
Nov 03 21:36:07.710879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.710822    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6219b\"/\"kube-root-ca.crt\""
Nov 03 21:36:07.713830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.713807    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:36:07.716195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716176    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.716268 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716210    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skm9m\" (UniqueName: \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.716375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716354    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.716441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716427    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.716488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716467    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.716537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.716497    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817176    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817231    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817273    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817322    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817356    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skm9m\" (UniqueName: \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817484    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817719    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817753    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.817857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.817780    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.819548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.819526    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.819749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.819732    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.876291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.876231    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skm9m\" (UniqueName: \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\") pod \"auth-test-model-a6219b-kserve-9d687fc-8fr58\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") " pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:07.926492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:07.926468    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:08.299390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:08.299354    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:36:08.300388 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:36:08.300354    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe8536c_0be4_4dad_aed8_9c8dc2b8ceae.slice/crio-760f4f2fd3f5d1e494c73fc59d688e46f43a54ef48468e650479921b4a95d87c WatchSource:0}: Error finding container 760f4f2fd3f5d1e494c73fc59d688e46f43a54ef48468e650479921b4a95d87c: Status 404 returned error can't find the container with id 760f4f2fd3f5d1e494c73fc59d688e46f43a54ef48468e650479921b4a95d87c
Nov 03 21:36:08.668096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:08.668013    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" event={"ID":"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae","Type":"ContainerStarted","Data":"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"}
Nov 03 21:36:08.668096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:08.668073    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" event={"ID":"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae","Type":"ContainerStarted","Data":"760f4f2fd3f5d1e494c73fc59d688e46f43a54ef48468e650479921b4a95d87c"}
Nov 03 21:36:08.783227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:08.783164    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" podStartSLOduration=1.783145647 podStartE2EDuration="1.783145647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:08.778168844 +0000 UTC m=+2870.409482321" watchObservedRunningTime="2026-11-03 21:36:08.783145647 +0000 UTC m=+2870.414459125"
Nov 03 21:36:08.927785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:08.927596    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:17.927699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:17.927666    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:17.940136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:17.940109    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:18.753249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:18.753221    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:36:19.262801 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:36:19.262768    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:36:20.551752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:20.551718    5533 scope.go:122] "RemoveContainer" containerID="16efa40c489b7cada68c978b09142be750620ae3d365bac23c64f404d9639b4b"
Nov 03 21:36:20.561538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:20.561515    5533 scope.go:122] "RemoveContainer" containerID="3153b65e5caf597f295c6b89809ed60699fa78575972e78250c96f197c6d36dd"
Nov 03 21:36:20.570574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:20.570545    5533 scope.go:122] "RemoveContainer" containerID="40fd00de6a396be35caf8a01a53cc2cf6640c1a557972fc535b6ee139b4099d6"
Nov 03 21:36:20.581178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:20.581158    5533 scope.go:122] "RemoveContainer" containerID="d1a380f5388745a7381a98b023d04fb365200d1f9e9cde0afb48452c836ab578"
Nov 03 21:36:30.585760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:30.585675    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:36:37.053030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:37.052995    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"]
Nov 03 21:36:37.099212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:37.099183    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a97d695-kgrd9"]
Nov 03 21:36:38.493938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:38.493908    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7ca452-bdd6-44c4-8c3b-ec848842be91" path="/var/lib/kubelet/pods/8d7ca452-bdd6-44c4-8c3b-ec848842be91/volumes"
Nov 03 21:36:40.257304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.257270    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:36:40.263474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.263452    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.318572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.318539    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:36:40.359259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.359216    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b22936\"/\"openshift-service-ca.crt\""
Nov 03 21:36:40.359259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.359228    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b22936\"/\"kube-root-ca.crt\""
Nov 03 21:36:40.359532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.359409    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b22936\"/\"default-dockercfg-rhsw8\""
Nov 03 21:36:40.359532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.359481    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b22936\"/\"explicit-model-b22936-kserve-self-signed-certs\""
Nov 03 21:36:40.375098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375065    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.375251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375126    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.375251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375155    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.375372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375280    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.375449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375400    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwpxk\" (UniqueName: \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.375449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.375422    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.476882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.476841    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwpxk\" (UniqueName: \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.476882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.476879    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.476942    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.476983    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.477018    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477154 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.477050    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.477359    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.477382    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.477505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.477421    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.479452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.479423    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.479593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.479573    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.534884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.534827    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwpxk\" (UniqueName: \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\") pod \"explicit-model-b22936-kserve-64686c86f5-fjt62\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") " pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.577744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.577720    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:40.950343 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:36:40.950313    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17390c66_0532_4e1a_8094_8810bed515eb.slice/crio-e99ca98466dacda82319fe8472f1e3cacb4d8efda222ca9895919bcb2302f7e5 WatchSource:0}: Error finding container e99ca98466dacda82319fe8472f1e3cacb4d8efda222ca9895919bcb2302f7e5: Status 404 returned error can't find the container with id e99ca98466dacda82319fe8472f1e3cacb4d8efda222ca9895919bcb2302f7e5
Nov 03 21:36:40.951241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:40.951209    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:36:41.928649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:41.928604    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" event={"ID":"17390c66-0532-4e1a-8094-8810bed515eb","Type":"ContainerStarted","Data":"998a5c5da461fcfff39a9b28b1ec311536ed7f3bb53e318d87a464985ca35fb1"}
Nov 03 21:36:41.929049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:41.928662    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" event={"ID":"17390c66-0532-4e1a-8094-8810bed515eb","Type":"ContainerStarted","Data":"e99ca98466dacda82319fe8472f1e3cacb4d8efda222ca9895919bcb2302f7e5"}
Nov 03 21:36:42.043740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:42.043680    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" podStartSLOduration=2.043663074 podStartE2EDuration="2.043663074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:42.039028117 +0000 UTC m=+2903.670341591" watchObservedRunningTime="2026-11-03 21:36:42.043663074 +0000 UTC m=+2903.674976557"
Nov 03 21:36:42.578112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:42.578068    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:49.263795 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:36:49.263753    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:36:50.578254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:50.578218    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:50.595387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:50.595363    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:36:51.006359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:36:51.006331    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:37:06.509326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.509293    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:37:06.513517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.513497    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.564830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.564799    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"auth-test-model-f0dc66-kserve-self-signed-certs\""
Nov 03 21:37:06.585459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585427    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.585628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585509    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.585628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585539    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8pw2\" (UniqueName: \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.585786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585749    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.585852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585821    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.585909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.585860    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.611303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.611280    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"default-dockercfg-fbt9z\""
Nov 03 21:37:06.611728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.611701    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"kube-root-ca.crt\""
Nov 03 21:37:06.611728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.611717    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"openshift-service-ca.crt\""
Nov 03 21:37:06.615023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.614997    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:37:06.687046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687011    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687092    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687111    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8pw2\" (UniqueName: \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687187    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687229    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687373    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687492    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.687828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.687804    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.688101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.688075    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.689510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.689485    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.690209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.690187    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.746112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.746087    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8pw2\" (UniqueName: \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\") pod \"auth-test-model-f0dc66-kserve-7f74c7497c-q687h\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:06.826878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:06.826822    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:07.201931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:07.201904    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:37:07.204100 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:37:07.204073    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30df6e44_f914_4747_af20_62e15d22c13d.slice/crio-7534bf5c08f4c5735eba3b72a461f11e565e5ddd29e180610c9b43b1de54326f WatchSource:0}: Error finding container 7534bf5c08f4c5735eba3b72a461f11e565e5ddd29e180610c9b43b1de54326f: Status 404 returned error can't find the container with id 7534bf5c08f4c5735eba3b72a461f11e565e5ddd29e180610c9b43b1de54326f
Nov 03 21:37:08.121831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:08.121794    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" event={"ID":"30df6e44-f914-4747-af20-62e15d22c13d","Type":"ContainerStarted","Data":"f2ffc4ea2c1bd0fa32aaf2f8227cbe05032886d8d76b8760a76166c1ce55e608"}
Nov 03 21:37:08.122216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:08.121844    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" event={"ID":"30df6e44-f914-4747-af20-62e15d22c13d","Type":"ContainerStarted","Data":"7534bf5c08f4c5735eba3b72a461f11e565e5ddd29e180610c9b43b1de54326f"}
Nov 03 21:37:08.234824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:08.234776    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" podStartSLOduration=2.234762746 podStartE2EDuration="2.234762746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:08.233379294 +0000 UTC m=+2929.864692772" watchObservedRunningTime="2026-11-03 21:37:08.234762746 +0000 UTC m=+2929.866076223"
Nov 03 21:37:08.827619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:08.827567    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:15.377655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:15.377606    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:37:15.378094 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:15.377855    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy" containerID="cri-o://4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a" gracePeriod=30
Nov 03 21:37:15.849654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:15.849606    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:37:15.849899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:15.849873    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy" containerID="cri-o://e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11" gracePeriod=30
Nov 03 21:37:16.827548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:16.827511    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:16.839565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:16.839541    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:17.212774 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:17.212743    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:37:19.263011 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:37:19.262974    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:37:20.648149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.648094    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:37:20.652728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.652623    5533 scope.go:122] "RemoveContainer" containerID="9a93d11928e38816146fe40ce8af8bd4d72088be074f23ef781f0ecc1b52652c"
Nov 03 21:37:20.739200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739171    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739214    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739245    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739292    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739379    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739494    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739543    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:37:20.739599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739584    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739705    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739735    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:20.739797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739752    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\") pod \"4f8463a1-788a-47c6-97ed-050a9a476698\" (UID: \"4f8463a1-788a-47c6-97ed-050a9a476698\") "
Nov 03 21:37:20.739797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739764    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:20.739971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.739945    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:20.740048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740028    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:20.740401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740382    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.740488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740405    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.740488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740420    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.740488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740432    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.740488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.740445    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8463a1-788a-47c6-97ed-050a9a476698-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.741539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.741499    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:20.741680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.741654    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:37:20.741827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.741808    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:20.741967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.741946    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js" pod "4f8463a1-788a-47c6-97ed-050a9a476698" (UID: "4f8463a1-788a-47c6-97ed-050a9a476698"). InnerVolumeSpecName "kube-api-access-rw2js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:20.842173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.842148    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8463a1-788a-47c6-97ed-050a9a476698-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.842173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.842171    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.842317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.842181    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8463a1-788a-47c6-97ed-050a9a476698-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:20.842317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:20.842210    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2js\" (UniqueName: \"kubernetes.io/projected/4f8463a1-788a-47c6-97ed-050a9a476698-kube-api-access-rw2js\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.108299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.108267    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:37:21.229725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.229659    5533 generic.go:356] "Generic (PLEG): container finished" podID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerID="e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11" exitCode=0
Nov 03 21:37:21.229725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.229696    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"
Nov 03 21:37:21.229930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.229735    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" event={"ID":"5e9387c8-4445-4592-899d-dea4674b0ab9","Type":"ContainerDied","Data":"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"}
Nov 03 21:37:21.229930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.229793    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd" event={"ID":"5e9387c8-4445-4592-899d-dea4674b0ab9","Type":"ContainerDied","Data":"936ae052319e5fe69745fe8774f9341056e060719c40365ba8c73a42db42ab45"}
Nov 03 21:37:21.229930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.229820    5533 scope.go:122] "RemoveContainer" containerID="e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"
Nov 03 21:37:21.231097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.231078    5533 generic.go:356] "Generic (PLEG): container finished" podID="4f8463a1-788a-47c6-97ed-050a9a476698" containerID="4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a" exitCode=0
Nov 03 21:37:21.231216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.231120    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"
Nov 03 21:37:21.231216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.231164    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" event={"ID":"4f8463a1-788a-47c6-97ed-050a9a476698","Type":"ContainerDied","Data":"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"}
Nov 03 21:37:21.231216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.231214    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5" event={"ID":"4f8463a1-788a-47c6-97ed-050a9a476698","Type":"ContainerDied","Data":"8057ede3c185d80a4210845139052c59b9acece7a253a799ddc7954aa5f0bcbe"}
Nov 03 21:37:21.240245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.240229    5533 scope.go:122] "RemoveContainer" containerID="e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"
Nov 03 21:37:21.240469 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:37:21.240450    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11\": container with ID starting with e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11 not found: ID does not exist" containerID="e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"
Nov 03 21:37:21.240512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.240483    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11"} err="failed to get container status \"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11\": rpc error: code = NotFound desc = could not find container \"e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11\": container with ID starting with e4b50dacd9605b16c8e349284a48f3c338101f45cca116fe27c48379febe2a11 not found: ID does not exist"
Nov 03 21:37:21.240512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.240504    5533 scope.go:122] "RemoveContainer" containerID="4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"
Nov 03 21:37:21.247835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.247814    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.247940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.247862    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.247940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.247904    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.247940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.247951    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.248156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.247982    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.248156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.248005    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.248156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.248066    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.248282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.248164    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.248282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.248210    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\") pod \"5e9387c8-4445-4592-899d-dea4674b0ab9\" (UID: \"5e9387c8-4445-4592-899d-dea4674b0ab9\") "
Nov 03 21:37:21.249048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.249025    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:21.249250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.249227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:21.250030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.249690    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:21.250030 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.249999    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:37:21.250276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.250195    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:21.250578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.250561    5533 scope.go:122] "RemoveContainer" containerID="4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"
Nov 03 21:37:21.253053 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:37:21.253027    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a\": container with ID starting with 4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a not found: ID does not exist" containerID="4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"
Nov 03 21:37:21.253140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.253062    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a"} err="failed to get container status \"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a\": rpc error: code = NotFound desc = could not find container \"4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a\": container with ID starting with 4780fe27ff024883ab88870b9ca72b4be86300d097d177f85fecfa9f5feeb89a not found: ID does not exist"
Nov 03 21:37:21.264067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.264039    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:37:21.264521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.264492    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:21.272028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.272003    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:21.276039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.276001    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2" pod "5e9387c8-4445-4592-899d-dea4674b0ab9" (UID: "5e9387c8-4445-4592-899d-dea4674b0ab9"). InnerVolumeSpecName "kube-api-access-g8sx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:21.350492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350465    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8sx2\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-kube-api-access-g8sx2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350492    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350501    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350511    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350519    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350529    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9387c8-4445-4592-899d-dea4674b0ab9-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350537    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350545    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.350749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.350552    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9387c8-4445-4592-899d-dea4674b0ab9-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:21.387255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.387225    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:37:21.391423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.391396    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-b85p5"]
Nov 03 21:37:21.646195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.646165    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:37:21.650611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:21.650577    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-pw5dd"]
Nov 03 21:37:22.496079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:22.496043    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" path="/var/lib/kubelet/pods/4f8463a1-788a-47c6-97ed-050a9a476698/volumes"
Nov 03 21:37:22.496528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:22.496513    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" path="/var/lib/kubelet/pods/5e9387c8-4445-4592-899d-dea4674b0ab9/volumes"
Nov 03 21:37:25.843374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.843339    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:37:25.843965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.843937    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy"
Nov 03 21:37:25.843965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.843959    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy"
Nov 03 21:37:25.844145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.843976    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy"
Nov 03 21:37:25.844145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.843981    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy"
Nov 03 21:37:25.844145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.844133    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9387c8-4445-4592-899d-dea4674b0ab9" containerName="istio-proxy"
Nov 03 21:37:25.844145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.844146    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8463a1-788a-47c6-97ed-050a9a476698" containerName="istio-proxy"
Nov 03 21:37:25.848944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.848928    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900042    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900098    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900182    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900226    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900320    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4z2m\" (UniqueName: \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.902663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.900447    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:25.946275 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.946252    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"openshift-service-ca.crt\""
Nov 03 21:37:25.946404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.946273    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"kube-root-ca.crt\""
Nov 03 21:37:25.946466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.946435    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"auth-test-model-4d1799-kserve-self-signed-certs\""
Nov 03 21:37:25.946542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.946522    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-4d1799\"/\"default-dockercfg-f75jv\""
Nov 03 21:37:25.949613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:25.949593    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:37:26.001764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.001235    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.001764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.001281    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.001764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.001329    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002048    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002148    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002311 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002290    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002344    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002437    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4z2m\" (UniqueName: \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.002952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.002896    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.008970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.008941    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.009243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.009219    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.063090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.063049    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4z2m\" (UniqueName: \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\") pod \"auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") " pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.161461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.161386    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:26.503190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.503156    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:37:26.536223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:26.536191    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:37:26.537278 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:37:26.537254    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0bb402b_6b0c_4fcd_9637_9b35f1153e3b.slice/crio-deb219d10a8856bb043831d3a7830ae4484b9853114acd4275e9be4aa65f1fe7 WatchSource:0}: Error finding container deb219d10a8856bb043831d3a7830ae4484b9853114acd4275e9be4aa65f1fe7: Status 404 returned error can't find the container with id deb219d10a8856bb043831d3a7830ae4484b9853114acd4275e9be4aa65f1fe7
Nov 03 21:37:27.125218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:27.125184    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:37:27.276880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:27.276844    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" event={"ID":"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b","Type":"ContainerStarted","Data":"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"}
Nov 03 21:37:27.277052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:27.276890    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" event={"ID":"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b","Type":"ContainerStarted","Data":"deb219d10a8856bb043831d3a7830ae4484b9853114acd4275e9be4aa65f1fe7"}
Nov 03 21:37:27.411839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:27.411724    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" podStartSLOduration=2.411710009 podStartE2EDuration="2.411710009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:27.409086019 +0000 UTC m=+2949.040399492" watchObservedRunningTime="2026-11-03 21:37:27.411710009 +0000 UTC m=+2949.043023485"
Nov 03 21:37:28.161492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:28.161461    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:36.162336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:36.162299    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:36.175345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:36.175320    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:36.356590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:36.356560    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:37:42.146044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.146004    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" podUID="17390c66-0532-4e1a-8094-8810bed515eb" containerName="main" containerID="cri-o://998a5c5da461fcfff39a9b28b1ec311536ed7f3bb53e318d87a464985ca35fb1" gracePeriod=45
Nov 03 21:37:42.394464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.394430    5533 generic.go:356] "Generic (PLEG): container finished" podID="17390c66-0532-4e1a-8094-8810bed515eb" containerID="998a5c5da461fcfff39a9b28b1ec311536ed7f3bb53e318d87a464985ca35fb1" exitCode=0
Nov 03 21:37:42.394690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.394671    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" event={"ID":"17390c66-0532-4e1a-8094-8810bed515eb","Type":"ContainerDied","Data":"998a5c5da461fcfff39a9b28b1ec311536ed7f3bb53e318d87a464985ca35fb1"}
Nov 03 21:37:42.427262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.427237    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:37:42.479399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479371    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.479549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479447    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.479549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479537    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.479718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479697    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:42.479786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479710    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.479847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479801    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:42.479903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479808    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.479903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479877    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:42.480024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.479898    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\" (UniqueName: \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\") pod \"17390c66-0532-4e1a-8094-8810bed515eb\" (UID: \"17390c66-0532-4e1a-8094-8810bed515eb\") "
Nov 03 21:37:42.480466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.480431    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:42.480466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.480458    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:42.480631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.480472    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:42.481866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.481840    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:37:42.481990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.481974    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "kube-api-access-vwpxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:37:42.482165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.482145    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm" pod "17390c66-0532-4e1a-8094-8810bed515eb" (UID: "17390c66-0532-4e1a-8094-8810bed515eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:37:42.582756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.582720    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17390c66-0532-4e1a-8094-8810bed515eb-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:42.582756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.582759    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwpxk\" (UniqueName: \"kubernetes.io/projected/17390c66-0532-4e1a-8094-8810bed515eb-kube-api-access-vwpxk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:42.582985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:42.582776    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17390c66-0532-4e1a-8094-8810bed515eb-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:37:43.403192 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:43.403159    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"
Nov 03 21:37:43.403685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:43.403199    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62" event={"ID":"17390c66-0532-4e1a-8094-8810bed515eb","Type":"ContainerDied","Data":"e99ca98466dacda82319fe8472f1e3cacb4d8efda222ca9895919bcb2302f7e5"}
Nov 03 21:37:43.403685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:43.403250    5533 scope.go:122] "RemoveContainer" containerID="998a5c5da461fcfff39a9b28b1ec311536ed7f3bb53e318d87a464985ca35fb1"
Nov 03 21:37:43.559972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:43.559941    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:37:43.568452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:43.568425    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b22936/explicit-model-b22936-kserve-64686c86f5-fjt62"]
Nov 03 21:37:44.494649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:44.494595    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17390c66-0532-4e1a-8094-8810bed515eb" path="/var/lib/kubelet/pods/17390c66-0532-4e1a-8094-8810bed515eb/volumes"
Nov 03 21:37:49.261131 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:37:49.261081    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:37:57.287445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:37:57.287148    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:38:02.288350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.288263    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" podUID="2b1097f4-d542-409f-b39f-3acc2c40c690" containerName="payload-pre-processing" containerID="cri-o://5798b3c3613cc2ee524babff35c6cd4639ceda3e98fa73f1189432cdf2994fbe" gracePeriod=25
Nov 03 21:38:02.550542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.550515    5533 generic.go:356] "Generic (PLEG): container finished" podID="2b1097f4-d542-409f-b39f-3acc2c40c690" containerID="5798b3c3613cc2ee524babff35c6cd4639ceda3e98fa73f1189432cdf2994fbe" exitCode=0
Nov 03 21:38:02.550701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.550592    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" event={"ID":"2b1097f4-d542-409f-b39f-3acc2c40c690","Type":"ContainerDied","Data":"5798b3c3613cc2ee524babff35c6cd4639ceda3e98fa73f1189432cdf2994fbe"}
Nov 03 21:38:02.550701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.550660    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck" event={"ID":"2b1097f4-d542-409f-b39f-3acc2c40c690","Type":"ContainerDied","Data":"25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf"}
Nov 03 21:38:02.550701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.550682    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25bc87b28c96092b53ff3e03ff07780a9e6b2c799184248942803dc8ec97cbaf"
Nov 03 21:38:02.564277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.564255    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:38:02.617068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.617044    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\") pod \"2b1097f4-d542-409f-b39f-3acc2c40c690\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") "
Nov 03 21:38:02.617205 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.617078    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\" (UniqueName: \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\") pod \"2b1097f4-d542-409f-b39f-3acc2c40c690\" (UID: \"2b1097f4-d542-409f-b39f-3acc2c40c690\") "
Nov 03 21:38:02.617413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.617392    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume" pod "2b1097f4-d542-409f-b39f-3acc2c40c690" (UID: "2b1097f4-d542-409f-b39f-3acc2c40c690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:38:02.617473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.617459    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1097f4-d542-409f-b39f-3acc2c40c690-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:02.619200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.619177    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr" pod "2b1097f4-d542-409f-b39f-3acc2c40c690" (UID: "2b1097f4-d542-409f-b39f-3acc2c40c690"). InnerVolumeSpecName "kube-api-access-nxqhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:02.718987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:02.718962    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqhr\" (UniqueName: \"kubernetes.io/projected/2b1097f4-d542-409f-b39f-3acc2c40c690-kube-api-access-nxqhr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:03.557473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:03.557441    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"
Nov 03 21:38:03.715677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:03.715650    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:38:03.720876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:03.720851    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b22936-69bc7c9db7-s9nck"]
Nov 03 21:38:04.493439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:04.493406    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1097f4-d542-409f-b39f-3acc2c40c690" path="/var/lib/kubelet/pods/2b1097f4-d542-409f-b39f-3acc2c40c690/volumes"
Nov 03 21:38:13.294890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:13.294857    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:38:19.054071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:19.054043    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:38:19.066905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:19.066880    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:38:19.090477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:19.090456    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:38:19.102932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:19.102912    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:38:19.260895 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:38:19.260862    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:38:20.684319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:20.684290    5533 scope.go:122] "RemoveContainer" containerID="60418aed24f9b97d4886ad33c8dd58b53ee437fcfa67cf71653306ecca502e11"
Nov 03 21:38:44.680188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:44.680152    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:38:44.682976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:44.680423    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy" containerID="cri-o://719b98a624140a8b4763eb05c10168a6c87cbad4ea8d38fb74ed2ee7efecca67" gracePeriod=30
Nov 03 21:38:49.260355 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:38:49.260317    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:38:49.895092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:49.895059    5533 generic.go:356] "Generic (PLEG): container finished" podID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerID="719b98a624140a8b4763eb05c10168a6c87cbad4ea8d38fb74ed2ee7efecca67" exitCode=0
Nov 03 21:38:49.895258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:49.895130    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" event={"ID":"db3ae28c-706c-4166-b11f-73bdaceeffe7","Type":"ContainerDied","Data":"719b98a624140a8b4763eb05c10168a6c87cbad4ea8d38fb74ed2ee7efecca67"}
Nov 03 21:38:49.943608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:49.943584    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:38:50.035253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035200    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035253 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035243    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035293    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035334    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035402    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035428    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035485    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035596    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\") pod \"db3ae28c-706c-4166-b11f-73bdaceeffe7\" (UID: \"db3ae28c-706c-4166-b11f-73bdaceeffe7\") "
Nov 03 21:38:50.035927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035481    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:38:50.035927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035765    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:38:50.035927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035740    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:38:50.035927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035812    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:38:50.035927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.035913    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:38:50.036194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.036136    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.036194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.036156    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.036194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.036173    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3ae28c-706c-4166-b11f-73bdaceeffe7-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.036194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.036188    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.036364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.036203    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.037729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.037704    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "kube-api-access-vgrb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:50.037858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.037796    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:50.038029 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.038013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:38:50.038092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.038066    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy" pod "db3ae28c-706c-4166-b11f-73bdaceeffe7" (UID: "db3ae28c-706c-4166-b11f-73bdaceeffe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:38:50.137001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.136978    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.137139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.137007    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.137139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.137017    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3ae28c-706c-4166-b11f-73bdaceeffe7-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.137139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.137028    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgrb2\" (UniqueName: \"kubernetes.io/projected/db3ae28c-706c-4166-b11f-73bdaceeffe7-kube-api-access-vgrb2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:50.902555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.902479    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"
Nov 03 21:38:50.902555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.902507    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m" event={"ID":"db3ae28c-706c-4166-b11f-73bdaceeffe7","Type":"ContainerDied","Data":"6ecd47a1226967fdc2c6fccf677ac37956d0762ef8af9930f4b0821d8ee70622"}
Nov 03 21:38:50.903059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:50.902576    5533 scope.go:122] "RemoveContainer" containerID="719b98a624140a8b4763eb05c10168a6c87cbad4ea8d38fb74ed2ee7efecca67"
Nov 03 21:38:51.060087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:51.060057    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:38:51.106847 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:51.106818    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-47j9m"]
Nov 03 21:38:52.495670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.495618    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" path="/var/lib/kubelet/pods/db3ae28c-706c-4166-b11f-73bdaceeffe7/volumes"
Nov 03 21:38:52.926817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.926787    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"]
Nov 03 21:38:52.927345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927330    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2b1097f4-d542-409f-b39f-3acc2c40c690" containerName="payload-pre-processing"
Nov 03 21:38:52.927401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927347    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1097f4-d542-409f-b39f-3acc2c40c690" containerName="payload-pre-processing"
Nov 03 21:38:52.927401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927367    5533 cpu_manager.go:414] "containerMap: removing container" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy"
Nov 03 21:38:52.927401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927372    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy"
Nov 03 21:38:52.927401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927392    5533 cpu_manager.go:414] "containerMap: removing container" podUID="17390c66-0532-4e1a-8094-8810bed515eb" containerName="main"
Nov 03 21:38:52.927401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927397    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17390c66-0532-4e1a-8094-8810bed515eb" containerName="main"
Nov 03 21:38:52.927596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927491    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1097f4-d542-409f-b39f-3acc2c40c690" containerName="payload-pre-processing"
Nov 03 21:38:52.927596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927501    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3ae28c-706c-4166-b11f-73bdaceeffe7" containerName="istio-proxy"
Nov 03 21:38:52.927596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.927512    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="17390c66-0532-4e1a-8094-8810bed515eb" containerName="main"
Nov 03 21:38:52.932074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.932055    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:52.982152 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:52.982124    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"]
Nov 03 21:38:53.024817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.024784    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:38:53.024817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.024813    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:38:53.066186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.066150    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.066337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.066306    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsqr\" (UniqueName: \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.066406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.066386    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.167699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.167631    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.167861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.167770    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsqr\" (UniqueName: \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.167861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.167811    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.168520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.168495    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.170486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.170453    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.225769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.225714    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsqr\" (UniqueName: \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\") pod \"maas-api-revoke-keys-e2e-explicit-b22936-8ntvx\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.245404 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.245380    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:53.619927 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:38:53.619898    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a25c50_1bd0_4b3e_8380_9fa06e16620c.slice/crio-16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe WatchSource:0}: Error finding container 16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe: Status 404 returned error can't find the container with id 16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe
Nov 03 21:38:53.620384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.620360    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"]
Nov 03 21:38:53.656899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.656875    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"]
Nov 03 21:38:53.662896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.662871    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.723521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.723499    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"]
Nov 03 21:38:53.774362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.774337    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.774483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.774392    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn66h\" (UniqueName: \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.774563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.774508    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.875630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.875576    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.875630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.875616    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn66h\" (UniqueName: \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.875836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.875656    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.876240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.876220    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.877758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.877741    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.933783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.933756    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn66h\" (UniqueName: \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\") pod \"maas-api-revoke-sub-bbb30a63-qn8tc\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:53.935733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.935711    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx" event={"ID":"88a25c50-1bd0-4b3e-8380-9fa06e16620c","Type":"ContainerStarted","Data":"2bcb0ad916049e3102d406f8bd052a1770cb720cabe1f7e704f0df0abc61f0da"}
Nov 03 21:38:53.935817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.935755    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx" event={"ID":"88a25c50-1bd0-4b3e-8380-9fa06e16620c","Type":"ContainerStarted","Data":"16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe"}
Nov 03 21:38:53.976572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:53.976548    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:54.046283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.046234    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx" podStartSLOduration=2.046220863 podStartE2EDuration="2.046220863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:54.044479652 +0000 UTC m=+3035.675793131" watchObservedRunningTime="2026-11-03 21:38:54.046220863 +0000 UTC m=+3035.677534340"
Nov 03 21:38:54.346749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.346719    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"]
Nov 03 21:38:54.347603 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:38:54.347568    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cddb9d6_5740_481a_9f07_503aca0ea8cf.slice/crio-52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90 WatchSource:0}: Error finding container 52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90: Status 404 returned error can't find the container with id 52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90
Nov 03 21:38:54.943669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.943628    5533 generic.go:356] "Generic (PLEG): container finished" podID="88a25c50-1bd0-4b3e-8380-9fa06e16620c" containerID="2bcb0ad916049e3102d406f8bd052a1770cb720cabe1f7e704f0df0abc61f0da" exitCode=0
Nov 03 21:38:54.944078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.943721    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx" event={"ID":"88a25c50-1bd0-4b3e-8380-9fa06e16620c","Type":"ContainerDied","Data":"2bcb0ad916049e3102d406f8bd052a1770cb720cabe1f7e704f0df0abc61f0da"}
Nov 03 21:38:54.945388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.945363    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc" event={"ID":"2cddb9d6-5740-481a-9f07-503aca0ea8cf","Type":"ContainerStarted","Data":"a6ce77d63f34704d7f31ee37c4efd43565f99e8f5f568c45e8b6988676d78896"}
Nov 03 21:38:54.945520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:54.945404    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc" event={"ID":"2cddb9d6-5740-481a-9f07-503aca0ea8cf","Type":"ContainerStarted","Data":"52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90"}
Nov 03 21:38:55.169277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:55.169228    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc" podStartSLOduration=2.169213516 podStartE2EDuration="2.169213516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:55.166623093 +0000 UTC m=+3036.797936571" watchObservedRunningTime="2026-11-03 21:38:55.169213516 +0000 UTC m=+3036.800526992"
Nov 03 21:38:55.953310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:55.953277    5533 generic.go:356] "Generic (PLEG): container finished" podID="2cddb9d6-5740-481a-9f07-503aca0ea8cf" containerID="a6ce77d63f34704d7f31ee37c4efd43565f99e8f5f568c45e8b6988676d78896" exitCode=0
Nov 03 21:38:55.953761 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:55.953367    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc" event={"ID":"2cddb9d6-5740-481a-9f07-503aca0ea8cf","Type":"ContainerDied","Data":"a6ce77d63f34704d7f31ee37c4efd43565f99e8f5f568c45e8b6988676d78896"}
Nov 03 21:38:56.094288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.094258    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:56.201920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.201898    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\") pod \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") "
Nov 03 21:38:56.202097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.201955    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\") pod \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") "
Nov 03 21:38:56.202097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.202031    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\" (UniqueName: \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\") pod \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\" (UID: \"88a25c50-1bd0-4b3e-8380-9fa06e16620c\") "
Nov 03 21:38:56.202373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.202337    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca" pod "88a25c50-1bd0-4b3e-8380-9fa06e16620c" (UID: "88a25c50-1bd0-4b3e-8380-9fa06e16620c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:38:56.202550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.202530    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:56.204520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.204452    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr" pod "88a25c50-1bd0-4b3e-8380-9fa06e16620c" (UID: "88a25c50-1bd0-4b3e-8380-9fa06e16620c"). InnerVolumeSpecName "kube-api-access-wpsqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:56.204520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.204761    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca" pod "88a25c50-1bd0-4b3e-8380-9fa06e16620c" (UID: "88a25c50-1bd0-4b3e-8380-9fa06e16620c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:38:56.303948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.303925    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a25c50-1bd0-4b3e-8380-9fa06e16620c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:56.303948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.303948    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsqr\" (UniqueName: \"kubernetes.io/projected/88a25c50-1bd0-4b3e-8380-9fa06e16620c-kube-api-access-wpsqr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:56.963087 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.963055    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"
Nov 03 21:38:56.963488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.963086    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx" event={"ID":"88a25c50-1bd0-4b3e-8380-9fa06e16620c","Type":"ContainerDied","Data":"16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe"}
Nov 03 21:38:56.963488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:56.963185    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16eb2898417c1dfc955bcdb6c5c55e1824f99c51e169457e1a699d718197aafe"
Nov 03 21:38:57.113090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.113059    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:57.213212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.213144    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\" (UniqueName: \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\") pod \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") "
Nov 03 21:38:57.213374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.213354    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\") pod \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") "
Nov 03 21:38:57.213418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.213409    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\") pod \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\" (UID: \"2cddb9d6-5740-481a-9f07-503aca0ea8cf\") "
Nov 03 21:38:57.213764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.213742    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca" pod "2cddb9d6-5740-481a-9f07-503aca0ea8cf" (UID: "2cddb9d6-5740-481a-9f07-503aca0ea8cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:38:57.213872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.213836    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:57.215293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.215268    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h" pod "2cddb9d6-5740-481a-9f07-503aca0ea8cf" (UID: "2cddb9d6-5740-481a-9f07-503aca0ea8cf"). InnerVolumeSpecName "kube-api-access-bn66h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:38:57.215435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.215319    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca" pod "2cddb9d6-5740-481a-9f07-503aca0ea8cf" (UID: "2cddb9d6-5740-481a-9f07-503aca0ea8cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:38:57.314675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.314622    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cddb9d6-5740-481a-9f07-503aca0ea8cf-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:57.314675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.314670    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn66h\" (UniqueName: \"kubernetes.io/projected/2cddb9d6-5740-481a-9f07-503aca0ea8cf-kube-api-access-bn66h\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:38:57.971484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.971457    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"
Nov 03 21:38:57.971880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.971492    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc" event={"ID":"2cddb9d6-5740-481a-9f07-503aca0ea8cf","Type":"ContainerDied","Data":"52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90"}
Nov 03 21:38:57.971880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:38:57.971540    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52cc0b44dfd471eb13fb353ff3ce3583e3f7a1171beb5524a5fd83b5c0afbc90"
Nov 03 21:39:04.429666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.429620    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"]
Nov 03 21:39:04.430324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430304    5533 cpu_manager.go:414] "containerMap: removing container" podUID="88a25c50-1bd0-4b3e-8380-9fa06e16620c" containerName="revoke-keys"
Nov 03 21:39:04.430392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430325    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a25c50-1bd0-4b3e-8380-9fa06e16620c" containerName="revoke-keys"
Nov 03 21:39:04.430392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430343    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2cddb9d6-5740-481a-9f07-503aca0ea8cf" containerName="revoke-keys"
Nov 03 21:39:04.430392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430349    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cddb9d6-5740-481a-9f07-503aca0ea8cf" containerName="revoke-keys"
Nov 03 21:39:04.430498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430450    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a25c50-1bd0-4b3e-8380-9fa06e16620c" containerName="revoke-keys"
Nov 03 21:39:04.430498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.430459    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cddb9d6-5740-481a-9f07-503aca0ea8cf" containerName="revoke-keys"
Nov 03 21:39:04.433987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.433970    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.488781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.488744    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.488943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.488874    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.488943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.488921    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.496494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.496453    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"]
Nov 03 21:39:04.532517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.532487    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:04.532710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.532592    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:39:04.590185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.590152    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.590362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.590196    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.590448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.590384    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.591025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.591001    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.593266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.593245    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.648495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.648469    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-e518e00c-sjjj6\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:04.747122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:04.747062    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:05.125334 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:05.125305    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"]
Nov 03 21:39:05.126953 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:05.126926    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1090cf3_7aeb_4ae7_824a_634127ab27b3.slice/crio-e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca WatchSource:0}: Error finding container e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca: Status 404 returned error can't find the container with id e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca
Nov 03 21:39:06.029697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:06.029659    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6" event={"ID":"b1090cf3-7aeb-4ae7-824a-634127ab27b3","Type":"ContainerStarted","Data":"e5d3a8349a0aa19e4f9cc604c814b043dfee7e13d8f94a47245f80323cc8ae95"}
Nov 03 21:39:06.030140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:06.029716    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6" event={"ID":"b1090cf3-7aeb-4ae7-824a-634127ab27b3","Type":"ContainerStarted","Data":"e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca"}
Nov 03 21:39:06.142881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:06.142836    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6" podStartSLOduration=2.142817325 podStartE2EDuration="2.142817325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:06.139459952 +0000 UTC m=+3047.770773426" watchObservedRunningTime="2026-11-03 21:39:06.142817325 +0000 UTC m=+3047.774130803"
Nov 03 21:39:07.037778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:07.037738    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1090cf3-7aeb-4ae7-824a-634127ab27b3" containerID="e5d3a8349a0aa19e4f9cc604c814b043dfee7e13d8f94a47245f80323cc8ae95" exitCode=0
Nov 03 21:39:07.038175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:07.037816    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6" event={"ID":"b1090cf3-7aeb-4ae7-824a-634127ab27b3","Type":"ContainerDied","Data":"e5d3a8349a0aa19e4f9cc604c814b043dfee7e13d8f94a47245f80323cc8ae95"}
Nov 03 21:39:07.157893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:07.157865    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:39:08.203340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.203313    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:08.233193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.233166    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\") pod \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") "
Nov 03 21:39:08.233317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.233246    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\") pod \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") "
Nov 03 21:39:08.233317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.233294    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\") pod \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\" (UID: \"b1090cf3-7aeb-4ae7-824a-634127ab27b3\") "
Nov 03 21:39:08.233705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.233679    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca" pod "b1090cf3-7aeb-4ae7-824a-634127ab27b3" (UID: "b1090cf3-7aeb-4ae7-824a-634127ab27b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:08.233846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.233830    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:08.235149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.235126    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca" pod "b1090cf3-7aeb-4ae7-824a-634127ab27b3" (UID: "b1090cf3-7aeb-4ae7-824a-634127ab27b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:08.235240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.235217    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss" pod "b1090cf3-7aeb-4ae7-824a-634127ab27b3" (UID: "b1090cf3-7aeb-4ae7-824a-634127ab27b3"). InnerVolumeSpecName "kube-api-access-wdpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:08.335188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.335122    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1090cf3-7aeb-4ae7-824a-634127ab27b3-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:08.335188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:08.335149    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/b1090cf3-7aeb-4ae7-824a-634127ab27b3-kube-api-access-wdpss\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:09.053700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:09.053673    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"
Nov 03 21:39:09.053700 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:09.053691    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6" event={"ID":"b1090cf3-7aeb-4ae7-824a-634127ab27b3","Type":"ContainerDied","Data":"e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca"}
Nov 03 21:39:09.053894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:09.053742    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b692ce43976a04cd9276a52280937d0397d2a5a3f96041f3eed5bcaa0ab8ca"
Nov 03 21:39:19.260711 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:19.260672    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:39:32.462661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.462602    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-16d904"]
Nov 03 21:39:32.464936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.463298    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1090cf3-7aeb-4ae7-824a-634127ab27b3" containerName="revoke-keys"
Nov 03 21:39:32.464936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.463318    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1090cf3-7aeb-4ae7-824a-634127ab27b3" containerName="revoke-keys"
Nov 03 21:39:32.464936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.463437    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1090cf3-7aeb-4ae7-824a-634127ab27b3" containerName="revoke-keys"
Nov 03 21:39:32.467694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.467668    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:32.510892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.510849    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwvx\" (UniqueName: \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\") pod \"test-curl-7490-16d904\" (UID: \"80b09857-a3fd-48a0-997e-64c1b3ddb1be\") " pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:32.537658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.525525    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 21:39:32.542678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.539707    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-16d904"]
Nov 03 21:39:32.561120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.561094    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 21:39:32.571721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.571692    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7vxf5\""
Nov 03 21:39:32.617741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.617707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwvx\" (UniqueName: \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\") pod \"test-curl-7490-16d904\" (UID: \"80b09857-a3fd-48a0-997e-64c1b3ddb1be\") " pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:32.677469 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.677438    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwvx\" (UniqueName: \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\") pod \"test-curl-7490-16d904\" (UID: \"80b09857-a3fd-48a0-997e-64c1b3ddb1be\") " pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:32.785425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:32.785343    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:33.154268 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:33.154241    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b09857_a3fd_48a0_997e_64c1b3ddb1be.slice/crio-1ac6fc5f5863fbbb580f5a3221bab8067d2bb37c81c69d4332af988f7f8c9786 WatchSource:0}: Error finding container 1ac6fc5f5863fbbb580f5a3221bab8067d2bb37c81c69d4332af988f7f8c9786: Status 404 returned error can't find the container with id 1ac6fc5f5863fbbb580f5a3221bab8067d2bb37c81c69d4332af988f7f8c9786
Nov 03 21:39:33.156130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:33.156073    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-16d904"]
Nov 03 21:39:33.157183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:33.157162    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:39:33.258319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:33.258277    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-16d904" event={"ID":"80b09857-a3fd-48a0-997e-64c1b3ddb1be","Type":"ContainerStarted","Data":"1ac6fc5f5863fbbb580f5a3221bab8067d2bb37c81c69d4332af988f7f8c9786"}
Nov 03 21:39:36.280983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:36.280942    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-16d904" event={"ID":"80b09857-a3fd-48a0-997e-64c1b3ddb1be","Type":"ContainerStarted","Data":"300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808"}
Nov 03 21:39:36.389969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:36.389918    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-16d904" podStartSLOduration=1.729176313 podStartE2EDuration="4.389905076s" totalImagesPullingTime="2.660728763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:36.387958274 +0000 UTC m=+3078.019271752" watchObservedRunningTime="2026-11-03 21:39:36.389905076 +0000 UTC m=+3078.021218552"
Nov 03 21:39:37.863482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:37.863444    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-16d904"]
Nov 03 21:39:37.868742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:37.868708    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-16d904"]
Nov 03 21:39:38.015732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.015705    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-84244e"]
Nov 03 21:39:38.016327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.016308    5533 cpu_manager.go:414] "containerMap: removing container" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" containerName="test-curl-7490-16d904"
Nov 03 21:39:38.016437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.016330    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" containerName="test-curl-7490-16d904"
Nov 03 21:39:38.016529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.016515    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" containerName="test-curl-7490-16d904"
Nov 03 21:39:38.020018 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.019999    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:38.072109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.072084    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-84244e"]
Nov 03 21:39:38.078437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.078413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkm6b\" (UniqueName: \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\") pod \"test-curl-7490-84244e\" (UID: \"897fa1fb-b0e4-4ecf-9f87-d705fe08b094\") " pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:38.179594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.179556    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkm6b\" (UniqueName: \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\") pod \"test-curl-7490-84244e\" (UID: \"897fa1fb-b0e4-4ecf-9f87-d705fe08b094\") " pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:38.237512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.237478    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkm6b\" (UniqueName: \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\") pod \"test-curl-7490-84244e\" (UID: \"897fa1fb-b0e4-4ecf-9f87-d705fe08b094\") " pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:38.295903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.295854    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-16d904" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" containerName="test-curl-7490-16d904" containerID="cri-o://300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808" gracePeriod=2
Nov 03 21:39:38.333080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.333052    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:38.343825 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:38.343799    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-16d904\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:38.538525 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:38.538448    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-16d904\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:38.705077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:38.705050    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-84244e"]
Nov 03 21:39:38.706116 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:38.706089    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod897fa1fb_b0e4_4ecf_9f87_d705fe08b094.slice/crio-2b8e4ec9dbc2ea9dd84eca75aa2eaab4770ace79a2c84fcc35878bad23e173e0 WatchSource:0}: Error finding container 2b8e4ec9dbc2ea9dd84eca75aa2eaab4770ace79a2c84fcc35878bad23e173e0: Status 404 returned error can't find the container with id 2b8e4ec9dbc2ea9dd84eca75aa2eaab4770ace79a2c84fcc35878bad23e173e0
Nov 03 21:39:39.305081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.305050    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-84244e" event={"ID":"897fa1fb-b0e4-4ecf-9f87-d705fe08b094","Type":"ContainerStarted","Data":"a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c"}
Nov 03 21:39:39.305464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.305090    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-84244e" event={"ID":"897fa1fb-b0e4-4ecf-9f87-d705fe08b094","Type":"ContainerStarted","Data":"2b8e4ec9dbc2ea9dd84eca75aa2eaab4770ace79a2c84fcc35878bad23e173e0"}
Nov 03 21:39:39.413067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.413012    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-84244e" podStartSLOduration=2.413000281 podStartE2EDuration="2.413000281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:39.411276217 +0000 UTC m=+3081.042589696" watchObservedRunningTime="2026-11-03 21:39:39.413000281 +0000 UTC m=+3081.044313757"
Nov 03 21:39:39.424142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.424115    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"]
Nov 03 21:39:39.428059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.428041    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.477214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.477174    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:39.477340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.477282    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:39:39.494146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.494117    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4tz\" (UniqueName: \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.494261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.494238    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.494351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.494336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.520489 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.520468    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"]
Nov 03 21:39:39.595538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.595475    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4tz\" (UniqueName: \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.595538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.595530    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.595737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.595574    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.596389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.596359    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.598022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.597993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.653212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.653150    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4tz\" (UniqueName: \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\") pod \"maas-api-revoke-sub-0edb5ae7-5rv5l\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:39.741601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:39.741576    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:40.132498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.132468    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"]
Nov 03 21:39:40.134079 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:40.134049    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af39d9_bbcd_4813_9dd6_7426152e959d.slice/crio-be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303 WatchSource:0}: Error finding container be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303: Status 404 returned error can't find the container with id be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303
Nov 03 21:39:40.312453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.312422    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l" event={"ID":"24af39d9-bbcd-4813-9dd6-7426152e959d","Type":"ContainerStarted","Data":"be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303"}
Nov 03 21:39:40.522491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.522465    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:40.607193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.607076    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\" (UniqueName: \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\") pod \"80b09857-a3fd-48a0-997e-64c1b3ddb1be\" (UID: \"80b09857-a3fd-48a0-997e-64c1b3ddb1be\") "
Nov 03 21:39:40.609563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.609536    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx" pod "80b09857-a3fd-48a0-997e-64c1b3ddb1be" (UID: "80b09857-a3fd-48a0-997e-64c1b3ddb1be"). InnerVolumeSpecName "kube-api-access-dlwvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:40.709041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.708984    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwvx\" (UniqueName: \"kubernetes.io/projected/80b09857-a3fd-48a0-997e-64c1b3ddb1be-kube-api-access-dlwvx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:40.973054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.972973    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-84244e"]
Nov 03 21:39:40.979075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:40.979038    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-84244e"]
Nov 03 21:39:41.320832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.320759    5533 generic.go:356] "Generic (PLEG): container finished" podID="24af39d9-bbcd-4813-9dd6-7426152e959d" containerID="a28040c7ffdd96c0e76c079ea604b7626d43ba7ba3a8fc786d8808e7fef4e1ef" exitCode=0
Nov 03 21:39:41.320832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.320810    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l" event={"ID":"24af39d9-bbcd-4813-9dd6-7426152e959d","Type":"ContainerDied","Data":"a28040c7ffdd96c0e76c079ea604b7626d43ba7ba3a8fc786d8808e7fef4e1ef"}
Nov 03 21:39:41.322177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.322156    5533 generic.go:356] "Generic (PLEG): container finished" podID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" containerID="300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808" exitCode=137
Nov 03 21:39:41.322297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.322190    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:41.322297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.322248    5533 scope.go:122] "RemoveContainer" containerID="300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808"
Nov 03 21:39:41.322493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.322470    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-84244e" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" containerName="test-curl-7490-84244e" containerID="cri-o://a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c" gracePeriod=2
Nov 03 21:39:41.332272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.332255    5533 scope.go:122] "RemoveContainer" containerID="300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808"
Nov 03 21:39:41.332522 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:41.332505    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808\": container with ID starting with 300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808 not found: ID does not exist" containerID="300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808"
Nov 03 21:39:41.332594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.332528    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808"} err="failed to get container status \"300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808\": rpc error: code = NotFound desc = could not find container \"300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808\": container with ID starting with 300321e5c0725fd438faecebebb38d0af1e6835f2d5bb7ca144a125ad05b4808 not found: ID does not exist"
Nov 03 21:39:41.362106 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.362082    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-44cfba"]
Nov 03 21:39:41.362958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.362940    5533 cpu_manager.go:414] "containerMap: removing container" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" containerName="test-curl-7490-84244e"
Nov 03 21:39:41.363017 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.362962    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" containerName="test-curl-7490-84244e"
Nov 03 21:39:41.363136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.363124    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" containerName="test-curl-7490-84244e"
Nov 03 21:39:41.368236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.368218    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:41.416453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.416428    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-44cfba"]
Nov 03 21:39:41.416823 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.416796    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnzs\" (UniqueName: \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\") pod \"test-curl-7490-44cfba\" (UID: \"3e601324-d97f-4e88-8d21-dbddabc803ce\") " pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:41.518098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.518059    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnzs\" (UniqueName: \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\") pod \"test-curl-7490-44cfba\" (UID: \"3e601324-d97f-4e88-8d21-dbddabc803ce\") " pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:41.575493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.575423    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnzs\" (UniqueName: \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\") pod \"test-curl-7490-44cfba\" (UID: \"3e601324-d97f-4e88-8d21-dbddabc803ce\") " pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:41.585847 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:41.585528    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-84244e\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:41.633296 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:41.633265    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-16d904\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:41.681269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:41.681246    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:42.047521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.047494    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-44cfba"]
Nov 03 21:39:42.048650 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:42.048609    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e601324_d97f_4e88_8d21_dbddabc803ce.slice/crio-d4e20546a46434ab884e4f07fc4adb03b119ccda8f42fe03e47ecdb44d08132e WatchSource:0}: Error finding container d4e20546a46434ab884e4f07fc4adb03b119ccda8f42fe03e47ecdb44d08132e: Status 404 returned error can't find the container with id d4e20546a46434ab884e4f07fc4adb03b119ccda8f42fe03e47ecdb44d08132e
Nov 03 21:39:42.331403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.331326    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-44cfba" event={"ID":"3e601324-d97f-4e88-8d21-dbddabc803ce","Type":"ContainerStarted","Data":"7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a"}
Nov 03 21:39:42.331403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.331365    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-44cfba" event={"ID":"3e601324-d97f-4e88-8d21-dbddabc803ce","Type":"ContainerStarted","Data":"d4e20546a46434ab884e4f07fc4adb03b119ccda8f42fe03e47ecdb44d08132e"}
Nov 03 21:39:42.442687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.442615    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-44cfba" podStartSLOduration=1.442596344 podStartE2EDuration="1.442596344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:42.437650784 +0000 UTC m=+3084.068964260" watchObservedRunningTime="2026-11-03 21:39:42.442596344 +0000 UTC m=+3084.073909827"
Nov 03 21:39:42.468486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.468462    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:42.485438 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:42.485406    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-84244e\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:42.494470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.494444    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" path="/var/lib/kubelet/pods/80b09857-a3fd-48a0-997e-64c1b3ddb1be/volumes"
Nov 03 21:39:42.529929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.529906    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\") pod \"24af39d9-bbcd-4813-9dd6-7426152e959d\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") "
Nov 03 21:39:42.530034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.529962    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\") pod \"24af39d9-bbcd-4813-9dd6-7426152e959d\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") "
Nov 03 21:39:42.530079 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.530045    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\" (UniqueName: \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\") pod \"24af39d9-bbcd-4813-9dd6-7426152e959d\" (UID: \"24af39d9-bbcd-4813-9dd6-7426152e959d\") "
Nov 03 21:39:42.530286 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.530266    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca" pod "24af39d9-bbcd-4813-9dd6-7426152e959d" (UID: "24af39d9-bbcd-4813-9dd6-7426152e959d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:42.530578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.530559    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:42.532024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.532001    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz" pod "24af39d9-bbcd-4813-9dd6-7426152e959d" (UID: "24af39d9-bbcd-4813-9dd6-7426152e959d"). InnerVolumeSpecName "kube-api-access-nj4tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:42.532084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.532041    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca" pod "24af39d9-bbcd-4813-9dd6-7426152e959d" (UID: "24af39d9-bbcd-4813-9dd6-7426152e959d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:42.533163 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:42.533136    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-16d904\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="80b09857-a3fd-48a0-997e-64c1b3ddb1be" pod="opendatahub/test-curl-7490-16d904"
Nov 03 21:39:42.631431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.631372    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj4tz\" (UniqueName: \"kubernetes.io/projected/24af39d9-bbcd-4813-9dd6-7426152e959d-kube-api-access-nj4tz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:42.631431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:42.631396    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24af39d9-bbcd-4813-9dd6-7426152e959d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:43.341007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.340986    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"
Nov 03 21:39:43.341325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.341025    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l" event={"ID":"24af39d9-bbcd-4813-9dd6-7426152e959d","Type":"ContainerDied","Data":"be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303"}
Nov 03 21:39:43.341325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.341111    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be735ceb2c05724c697ea19dd4e79049fbb8da9ef2e16153ff1edb553399f303"
Nov 03 21:39:43.582283 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.582222    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:43.640549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.640525    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\" (UniqueName: \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\") pod \"897fa1fb-b0e4-4ecf-9f87-d705fe08b094\" (UID: \"897fa1fb-b0e4-4ecf-9f87-d705fe08b094\") "
Nov 03 21:39:43.642524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.642498    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b" pod "897fa1fb-b0e4-4ecf-9f87-d705fe08b094" (UID: "897fa1fb-b0e4-4ecf-9f87-d705fe08b094"). InnerVolumeSpecName "kube-api-access-bkm6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:43.741773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.741744    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkm6b\" (UniqueName: \"kubernetes.io/projected/897fa1fb-b0e4-4ecf-9f87-d705fe08b094-kube-api-access-bkm6b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:43.774718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.774689    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:39:43.774940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.774917    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" containerID="cri-o://e78420ad35a3831fbc2cb642b1a0f748cf891fa4eacfae7807731b8a21979dd0" gracePeriod=30
Nov 03 21:39:43.878877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.878808    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-44cfba"]
Nov 03 21:39:43.884114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:43.884087    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-44cfba"]
Nov 03 21:39:44.033901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.033866    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7490-aa9207"]
Nov 03 21:39:44.034480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034458    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" containerName="test-curl-7490-44cfba"
Nov 03 21:39:44.034480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034478    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" containerName="test-curl-7490-44cfba"
Nov 03 21:39:44.034623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034504    5533 cpu_manager.go:414] "containerMap: removing container" podUID="24af39d9-bbcd-4813-9dd6-7426152e959d" containerName="revoke-keys"
Nov 03 21:39:44.034623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034510    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24af39d9-bbcd-4813-9dd6-7426152e959d" containerName="revoke-keys"
Nov 03 21:39:44.034623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034610    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" containerName="test-curl-7490-44cfba"
Nov 03 21:39:44.034797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.034624    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="24af39d9-bbcd-4813-9dd6-7426152e959d" containerName="revoke-keys"
Nov 03 21:39:44.038147 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.038132    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:44.092022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.091991    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-aa9207"]
Nov 03 21:39:44.146678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.146587    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wvv\" (UniqueName: \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\") pod \"test-curl-7490-aa9207\" (UID: \"26568124-d80b-44f6-857a-7bc10c95eea7\") " pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:44.248178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.248151    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wvv\" (UniqueName: \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\") pod \"test-curl-7490-aa9207\" (UID: \"26568124-d80b-44f6-857a-7bc10c95eea7\") " pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:44.306096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.306067    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wvv\" (UniqueName: \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\") pod \"test-curl-7490-aa9207\" (UID: \"26568124-d80b-44f6-857a-7bc10c95eea7\") " pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:44.353227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.353196    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:44.353693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.353228    5533 generic.go:356] "Generic (PLEG): container finished" podID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" containerID="a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c" exitCode=137
Nov 03 21:39:44.353693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.353282    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:44.353693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.353341    5533 scope.go:122] "RemoveContainer" containerID="a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c"
Nov 03 21:39:44.353693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.353492    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-44cfba" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" containerName="test-curl-7490-44cfba" containerID="cri-o://7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a" gracePeriod=2
Nov 03 21:39:44.373284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.373152    5533 scope.go:122] "RemoveContainer" containerID="a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c"
Nov 03 21:39:44.373488 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:44.373459    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c\": container with ID starting with a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c not found: ID does not exist" containerID="a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c"
Nov 03 21:39:44.373544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.373503    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c"} err="failed to get container status \"a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c\": rpc error: code = NotFound desc = could not find container \"a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c\": container with ID starting with a2d1e4017b844912cd77bfe3fb33fc6647776f5faad29f43688a6dc7f3529b9c not found: ID does not exist"
Nov 03 21:39:44.401555 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:44.401415    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-44cfba\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:44.449603 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:44.449557    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-44cfba\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:44.494804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.494768    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" path="/var/lib/kubelet/pods/897fa1fb-b0e4-4ecf-9f87-d705fe08b094/volumes"
Nov 03 21:39:44.497488 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:44.497455    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-84244e\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="897fa1fb-b0e4-4ecf-9f87-d705fe08b094" pod="opendatahub/test-curl-7490-84244e"
Nov 03 21:39:44.732896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:44.732865    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7490-aa9207"]
Nov 03 21:39:44.733996 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:44.733969    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26568124_d80b_44f6_857a_7bc10c95eea7.slice/crio-a07153afe472678f19c4fb00d4cb9917d76138b1943289dcb9ee1e633594b6bb WatchSource:0}: Error finding container a07153afe472678f19c4fb00d4cb9917d76138b1943289dcb9ee1e633594b6bb: Status 404 returned error can't find the container with id a07153afe472678f19c4fb00d4cb9917d76138b1943289dcb9ee1e633594b6bb
Nov 03 21:39:45.364428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:45.364392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-aa9207" event={"ID":"26568124-d80b-44f6-857a-7bc10c95eea7","Type":"ContainerStarted","Data":"8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6"}
Nov 03 21:39:45.364909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:45.364444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7490-aa9207" event={"ID":"26568124-d80b-44f6-857a-7bc10c95eea7","Type":"ContainerStarted","Data":"a07153afe472678f19c4fb00d4cb9917d76138b1943289dcb9ee1e633594b6bb"}
Nov 03 21:39:45.473372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:45.473309    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7490-aa9207" podStartSLOduration=2.473291213 podStartE2EDuration="2.473291213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:45.470883588 +0000 UTC m=+3087.102197066" watchObservedRunningTime="2026-11-03 21:39:45.473291213 +0000 UTC m=+3087.104604692"
Nov 03 21:39:45.613776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:45.613744    5533 patch_prober.go:29] interesting pod/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:39:45.613961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:45.613805    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:39:46.589132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.589101    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:46.674897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.674874    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\" (UniqueName: \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\") pod \"3e601324-d97f-4e88-8d21-dbddabc803ce\" (UID: \"3e601324-d97f-4e88-8d21-dbddabc803ce\") "
Nov 03 21:39:46.676751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.676726    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs" pod "3e601324-d97f-4e88-8d21-dbddabc803ce" (UID: "3e601324-d97f-4e88-8d21-dbddabc803ce"). InnerVolumeSpecName "kube-api-access-cjnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:46.776330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.776296    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnzs\" (UniqueName: \"kubernetes.io/projected/3e601324-d97f-4e88-8d21-dbddabc803ce-kube-api-access-cjnzs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:46.976341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.975737    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7490-aa9207"]
Nov 03 21:39:46.979829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:46.979569    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7490-aa9207"]
Nov 03 21:39:47.380735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.380660    5533 generic.go:356] "Generic (PLEG): container finished" podID="3e601324-d97f-4e88-8d21-dbddabc803ce" containerID="7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a" exitCode=137
Nov 03 21:39:47.380896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.380767    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:47.380896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.380775    5533 scope.go:122] "RemoveContainer" containerID="7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a"
Nov 03 21:39:47.381047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.381027    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7490-aa9207" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" containerName="test-curl-7490-aa9207" containerID="cri-o://8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6" gracePeriod=2
Nov 03 21:39:47.391266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.391249    5533 scope.go:122] "RemoveContainer" containerID="7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a"
Nov 03 21:39:47.391539 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:47.391519    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a\": container with ID starting with 7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a not found: ID does not exist" containerID="7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a"
Nov 03 21:39:47.391608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:47.391545    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a"} err="failed to get container status \"7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a\": rpc error: code = NotFound desc = could not find container \"7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a\": container with ID starting with 7ac6c0e1416794b75773cec896cdcaec83085ff17d069ae4f697b9957cb44a0a not found: ID does not exist"
Nov 03 21:39:47.429097 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:47.429066    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-aa9207\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:47.476958 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:47.476922    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-44cfba\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:47.524686 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:47.524651    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-aa9207\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:48.493242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:48.493210    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" path="/var/lib/kubelet/pods/3e601324-d97f-4e88-8d21-dbddabc803ce/volumes"
Nov 03 21:39:48.537968 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:48.537936    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-44cfba\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="3e601324-d97f-4e88-8d21-dbddabc803ce" pod="opendatahub/test-curl-7490-44cfba"
Nov 03 21:39:48.585789 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:48.585753    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-aa9207\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:49.260174 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:49.260144    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:39:49.398882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.398856    5533 generic.go:356] "Generic (PLEG): container finished" podID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerID="e78420ad35a3831fbc2cb642b1a0f748cf891fa4eacfae7807731b8a21979dd0" exitCode=0
Nov 03 21:39:49.399023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.398928    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" event={"ID":"74b44d7b-7466-44c0-a4db-84ff6b66a21b","Type":"ContainerDied","Data":"e78420ad35a3831fbc2cb642b1a0f748cf891fa4eacfae7807731b8a21979dd0"}
Nov 03 21:39:49.468812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.468788    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:39:49.612086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612064    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612161    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612202    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612245    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612306    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612330    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612356    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612380    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612419    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:49.612471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612457    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\") pod \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\" (UID: \"74b44d7b-7466-44c0-a4db-84ff6b66a21b\") "
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612560    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612755    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612784    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.612991    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.613009    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.613021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.613019    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.613388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.613028    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.613388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.613037    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.614610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.614580    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:39:49.614785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.614757    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "kube-api-access-nd6t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:49.614905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.614823    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:49.614905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.614894    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:49.615015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.614901    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy" pod "74b44d7b-7466-44c0-a4db-84ff6b66a21b" (UID: "74b44d7b-7466-44c0-a4db-84ff6b66a21b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:39:49.713758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.713727    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\" (UniqueName: \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\") pod \"26568124-d80b-44f6-857a-7bc10c95eea7\" (UID: \"26568124-d80b-44f6-857a-7bc10c95eea7\") "
Nov 03 21:39:49.714056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.714037    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6t8\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-kube-api-access-nd6t8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.714056 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.714054    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.714204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.714064    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.714204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.714073    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b44d7b-7466-44c0-a4db-84ff6b66a21b-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:49.715692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.715671    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv" pod "26568124-d80b-44f6-857a-7bc10c95eea7" (UID: "26568124-d80b-44f6-857a-7bc10c95eea7"). InnerVolumeSpecName "kube-api-access-x7wvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:49.815305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:49.815277    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wvv\" (UniqueName: \"kubernetes.io/projected/26568124-d80b-44f6-857a-7bc10c95eea7-kube-api-access-x7wvv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:50.130569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.130486    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"]
Nov 03 21:39:50.131211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131190    5533 cpu_manager.go:414] "containerMap: removing container" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" containerName="test-curl-7490-aa9207"
Nov 03 21:39:50.131211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131211    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" containerName="test-curl-7490-aa9207"
Nov 03 21:39:50.131352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131220    5533 cpu_manager.go:414] "containerMap: removing container" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy"
Nov 03 21:39:50.131352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131225    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy"
Nov 03 21:39:50.131352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131307    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" containerName="test-curl-7490-aa9207"
Nov 03 21:39:50.131352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.131315    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" containerName="istio-proxy"
Nov 03 21:39:50.135314 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.135294    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.184755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.184733    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:39:50.185084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.184986    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:39:50.189748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.189724    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"]
Nov 03 21:39:50.220401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.220374    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.220513 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.220423    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nfk\" (UniqueName: \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.220565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.220520    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.321099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.321075    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.321212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.321203    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.321284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.321227    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nfk\" (UniqueName: \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.321805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.321782    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.323391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.323370    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.383997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.383948    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nfk\" (UniqueName: \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\") pod \"maas-api-revoke-sub-74094f44-rsg2j\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.407650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.407618    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"
Nov 03 21:39:50.407759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.407662    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd" event={"ID":"74b44d7b-7466-44c0-a4db-84ff6b66a21b","Type":"ContainerDied","Data":"bb193d2bdfe4ef59a0b7b8c9af3e9be1e32ac90d6f1a38459200da9ba722dea0"}
Nov 03 21:39:50.407759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.407720    5533 scope.go:122] "RemoveContainer" containerID="e78420ad35a3831fbc2cb642b1a0f748cf891fa4eacfae7807731b8a21979dd0"
Nov 03 21:39:50.409226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.409205    5533 generic.go:356] "Generic (PLEG): container finished" podID="26568124-d80b-44f6-857a-7bc10c95eea7" containerID="8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6" exitCode=137
Nov 03 21:39:50.409327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.409247    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:50.418501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.418484    5533 scope.go:122] "RemoveContainer" containerID="8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6"
Nov 03 21:39:50.427652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.427624    5533 scope.go:122] "RemoveContainer" containerID="8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6"
Nov 03 21:39:50.427927 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:50.427908    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6\": container with ID starting with 8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6 not found: ID does not exist" containerID="8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6"
Nov 03 21:39:50.427985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.427934    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6"} err="failed to get container status \"8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6\": rpc error: code = NotFound desc = could not find container \"8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6\": container with ID starting with 8a1ca91082a18151ee9625cab9c85c20ea5860323e6d3e6a896a8f3937bc6dd6 not found: ID does not exist"
Nov 03 21:39:50.452267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.452226    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:50.494768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.494743    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" path="/var/lib/kubelet/pods/26568124-d80b-44f6-857a-7bc10c95eea7/volumes"
Nov 03 21:39:50.578951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.578916    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:39:50.581809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.581782    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e84357-openshift-default-6ff98f5885-k4mbd"]
Nov 03 21:39:50.624619 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:39:50.624586    5533 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7490-aa9207\" is forbidden: User \"system:node:ip-10-0-140-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.us-west-2.compute.internal' and this object" podUID="26568124-d80b-44f6-857a-7bc10c95eea7" pod="opendatahub/test-curl-7490-aa9207"
Nov 03 21:39:50.822220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:50.822193    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"]
Nov 03 21:39:50.822570 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:50.822549    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f433f_0632_479f_8bed_c4409f3f2194.slice/crio-be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf WatchSource:0}: Error finding container be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf: Status 404 returned error can't find the container with id be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf
Nov 03 21:39:51.419220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:51.419146    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j" event={"ID":"548f433f-0632-479f-8bed-c4409f3f2194","Type":"ContainerStarted","Data":"8b1e7e61a4c4ac13f2e66f7182f5fdb1b28ab07d267274f309e904c988f091b0"}
Nov 03 21:39:51.419220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:51.419206    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j" event={"ID":"548f433f-0632-479f-8bed-c4409f3f2194","Type":"ContainerStarted","Data":"be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf"}
Nov 03 21:39:51.530504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:51.530456    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j" podStartSLOduration=1.530442289 podStartE2EDuration="1.530442289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:51.527443301 +0000 UTC m=+3093.158756777" watchObservedRunningTime="2026-11-03 21:39:51.530442289 +0000 UTC m=+3093.161755806"
Nov 03 21:39:52.429098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:52.429065    5533 generic.go:356] "Generic (PLEG): container finished" podID="548f433f-0632-479f-8bed-c4409f3f2194" containerID="8b1e7e61a4c4ac13f2e66f7182f5fdb1b28ab07d267274f309e904c988f091b0" exitCode=0
Nov 03 21:39:52.429528 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:52.429105    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j" event={"ID":"548f433f-0632-479f-8bed-c4409f3f2194","Type":"ContainerDied","Data":"8b1e7e61a4c4ac13f2e66f7182f5fdb1b28ab07d267274f309e904c988f091b0"}
Nov 03 21:39:52.493954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:52.493920    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b44d7b-7466-44c0-a4db-84ff6b66a21b" path="/var/lib/kubelet/pods/74b44d7b-7466-44c0-a4db-84ff6b66a21b/volumes"
Nov 03 21:39:53.583609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.583587    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:53.656605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.656567    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\") pod \"548f433f-0632-479f-8bed-c4409f3f2194\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") "
Nov 03 21:39:53.656782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.656738    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\") pod \"548f433f-0632-479f-8bed-c4409f3f2194\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") "
Nov 03 21:39:53.656837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.656793    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\" (UniqueName: \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\") pod \"548f433f-0632-479f-8bed-c4409f3f2194\" (UID: \"548f433f-0632-479f-8bed-c4409f3f2194\") "
Nov 03 21:39:53.656951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.656912    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca" pod "548f433f-0632-479f-8bed-c4409f3f2194" (UID: "548f433f-0632-479f-8bed-c4409f3f2194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:39:53.657274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.657249    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:53.659134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.659103    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca" pod "548f433f-0632-479f-8bed-c4409f3f2194" (UID: "548f433f-0632-479f-8bed-c4409f3f2194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:39:53.659418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.659397    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk" pod "548f433f-0632-479f-8bed-c4409f3f2194" (UID: "548f433f-0632-479f-8bed-c4409f3f2194"). InnerVolumeSpecName "kube-api-access-l5nfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:53.758232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.758158    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f433f-0632-479f-8bed-c4409f3f2194-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:53.758232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:53.758192    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nfk\" (UniqueName: \"kubernetes.io/projected/548f433f-0632-479f-8bed-c4409f3f2194-kube-api-access-l5nfk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:39:54.443500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:54.443471    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"
Nov 03 21:39:54.443500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:54.443486    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j" event={"ID":"548f433f-0632-479f-8bed-c4409f3f2194","Type":"ContainerDied","Data":"be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf"}
Nov 03 21:39:54.443741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:54.443534    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2c866e8dacbab0a1948290b07b03bc2ce25c1601ec92c619cb75a25cbef0bf"
Nov 03 21:39:55.253098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.253063    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:39:55.253753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.253735    5533 cpu_manager.go:414] "containerMap: removing container" podUID="548f433f-0632-479f-8bed-c4409f3f2194" containerName="revoke-keys"
Nov 03 21:39:55.253753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.253754    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548f433f-0632-479f-8bed-c4409f3f2194" containerName="revoke-keys"
Nov 03 21:39:55.253893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.253883    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="548f433f-0632-479f-8bed-c4409f3f2194" containerName="revoke-keys"
Nov 03 21:39:55.257335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.257320    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.306614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.306528    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-130a8c-openshift-default-dockercfg-2wf7s\""
Nov 03 21:39:55.359656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.358874    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:39:55.376337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376298    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376357    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376450    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bqq\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376496    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376643    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376685    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376754    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376814    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.376872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.376840    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477521 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477495    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477537    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477575    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477605    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477627    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477739    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477789    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.477901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477867    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bqq\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.477932    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.478004    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.478076    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.478151    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478353 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.478198    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.478780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.478758    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.480089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.480067    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.480241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.480222    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.535450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.535397    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bqq\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.535744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.535728    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\") pod \"e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") " pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.570552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.570302    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:55.995378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.995345    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:39:55.996506 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:39:55.996476    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168a1c45_db66_4eb3_8327_978472d5538d.slice/crio-5b3a21007cfca814408b2fad6e68a878ec286a15076ad7285545af713f0a7ef4 WatchSource:0}: Error finding container 5b3a21007cfca814408b2fad6e68a878ec286a15076ad7285545af713f0a7ef4: Status 404 returned error can't find the container with id 5b3a21007cfca814408b2fad6e68a878ec286a15076ad7285545af713f0a7ef4
Nov 03 21:39:55.999031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.998998    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:39:55.999118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.999061    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:39:55.999118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:55.999103    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:39:56.459115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:56.459078    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerStarted","Data":"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"}
Nov 03 21:39:56.459502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:56.459135    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerStarted","Data":"5b3a21007cfca814408b2fad6e68a878ec286a15076ad7285545af713f0a7ef4"}
Nov 03 21:39:56.572375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:56.572349    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:56.575968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:56.575920    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podStartSLOduration=1.5759066480000001 podStartE2EDuration="1.575906648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:56.572812238 +0000 UTC m=+3098.204125715" watchObservedRunningTime="2026-11-03 21:39:56.575906648 +0000 UTC m=+3098.207220125"
Nov 03 21:39:57.578181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:57.578144    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:58.474254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:58.474225    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:39:58.475059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:39:58.475033    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:40:05.508744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:05.508710    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"]
Nov 03 21:40:05.553882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:05.553851    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b22936-8ntvx"]
Nov 03 21:40:06.497178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:06.497143    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a25c50-1bd0-4b3e-8380-9fa06e16620c" path="/var/lib/kubelet/pods/88a25c50-1bd0-4b3e-8380-9fa06e16620c/volumes"
Nov 03 21:40:06.981240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:06.981201    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:40:06.990940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:06.990913    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.043942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.043907    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-21475a-openshift-default-dockercfg-4zs8l\""
Nov 03 21:40:07.104356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.104323    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:40:07.108630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108596    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.108810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108693    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.108810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108721    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.108810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108793    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.109000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108828    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndm2f\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.109000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108862    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.109000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.108909    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.109164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.109036    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.109164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.109128    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210031    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210123    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210173    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210200    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210223    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210288    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210320    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndm2f\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210354    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210403    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210624    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210839    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210857    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.210987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.210968    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.211051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.211028    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.212755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.212733    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.212991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.212971    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.267597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.267542    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.268064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.268041    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndm2f\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\") pod \"e2e-noten-21475a-openshift-default-678c5f894f-pzpwf\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") " pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.306574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.306306    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:07.731466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.731438    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:40:07.732401 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:40:07.732369    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088d43cc_61c0_475f_81ad_dc85454a08a7.slice/crio-441e3e348929376a5f3ab1f775f71205eed0024adc842212fca70a952de08904 WatchSource:0}: Error finding container 441e3e348929376a5f3ab1f775f71205eed0024adc842212fca70a952de08904: Status 404 returned error can't find the container with id 441e3e348929376a5f3ab1f775f71205eed0024adc842212fca70a952de08904
Nov 03 21:40:07.734963 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.734929    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:07.735058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.734991    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:07.735058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:07.735021    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:08.564990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:08.564945    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" event={"ID":"088d43cc-61c0-475f-81ad-dc85454a08a7","Type":"ContainerStarted","Data":"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"}
Nov 03 21:40:08.565388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:08.565013    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" event={"ID":"088d43cc-61c0-475f-81ad-dc85454a08a7","Type":"ContainerStarted","Data":"441e3e348929376a5f3ab1f775f71205eed0024adc842212fca70a952de08904"}
Nov 03 21:40:08.681323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:08.681275    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" podStartSLOduration=2.681259996 podStartE2EDuration="2.681259996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:08.678518596 +0000 UTC m=+3110.309832072" watchObservedRunningTime="2026-11-03 21:40:08.681259996 +0000 UTC m=+3110.312573472"
Nov 03 21:40:09.307936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:09.307905    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:09.313585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:09.313557    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:09.573058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:09.572980    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:09.573953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:09.573930    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:40:12.441996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:12.441961    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:40:19.261945 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:40:19.261908    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:40:20.741408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:20.741373    5533 scope.go:122] "RemoveContainer" containerID="5798b3c3613cc2ee524babff35c6cd4639ceda3e98fa73f1189432cdf2994fbe"
Nov 03 21:40:32.355216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:32.355183    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:40:32.355854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:32.355453    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" containerID="cri-o://a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7" gracePeriod=30
Nov 03 21:40:36.413589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:36.413544    5533 patch_prober.go:29] interesting pod/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:40:36.414011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:36.413619    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:40:37.621905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.621844    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:40:37.774290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774260    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774316    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774352    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774394    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774437    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774494 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774487    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774581    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774611    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774741    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:40:37.774805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774746    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\") pod \"f504460b-0099-4ecb-a8f8-e3493bae4258\" (UID: \"f504460b-0099-4ecb-a8f8-e3493bae4258\") "
Nov 03 21:40:37.774805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.774777    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:40:37.775112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775028    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:40:37.775112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775043    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:37.775267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775244    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:40:37.775701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775589    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.775701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775619    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.775701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775648    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f504460b-0099-4ecb-a8f8-e3493bae4258-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.775701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775663    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.775701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.775678    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.776967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.776941    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:37.777093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.776972    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:40:37.777093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.776995    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "kube-api-access-2nfbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:37.777299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777270    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy" pod "f504460b-0099-4ecb-a8f8-e3493bae4258" (UID: "f504460b-0099-4ecb-a8f8-e3493bae4258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:40:37.777820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777799    5533 generic.go:356] "Generic (PLEG): container finished" podID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerID="a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7" exitCode=0
Nov 03 21:40:37.777918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777848    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"
Nov 03 21:40:37.777918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777853    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" event={"ID":"f504460b-0099-4ecb-a8f8-e3493bae4258","Type":"ContainerDied","Data":"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"}
Nov 03 21:40:37.777918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777908    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2" event={"ID":"f504460b-0099-4ecb-a8f8-e3493bae4258","Type":"ContainerDied","Data":"35e6a650592d94b203a801ecee47062b1d6401da0b81af639d46e67c05c191db"}
Nov 03 21:40:37.778095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.777944    5533 scope.go:122] "RemoveContainer" containerID="a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"
Nov 03 21:40:37.809065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.809035    5533 scope.go:122] "RemoveContainer" containerID="a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"
Nov 03 21:40:37.809395 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:40:37.809367    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7\": container with ID starting with a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7 not found: ID does not exist" containerID="a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"
Nov 03 21:40:37.809509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.809407    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7"} err="failed to get container status \"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7\": rpc error: code = NotFound desc = could not find container \"a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7\": container with ID starting with a4f8bd2381692f0101603954b7bb06d7835cab23d868fee8570d08aca617ded7 not found: ID does not exist"
Nov 03 21:40:37.876757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.876695    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.876757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.876721    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.876757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.876733    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f504460b-0099-4ecb-a8f8-e3493bae4258-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.876757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.876742    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfbf\" (UniqueName: \"kubernetes.io/projected/f504460b-0099-4ecb-a8f8-e3493bae4258-kube-api-access-2nfbf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:37.940908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.940877    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:40:37.943933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:37.943908    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-791c32-openshift-default-7f4cbf5995-52vk2"]
Nov 03 21:40:38.495072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:38.495040    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" path="/var/lib/kubelet/pods/f504460b-0099-4ecb-a8f8-e3493bae4258/volumes"
Nov 03 21:40:49.261712 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:40:49.261672    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:40:58.817892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.817854    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:40:58.828390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.828357    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy"
Nov 03 21:40:58.828390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.828389    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy"
Nov 03 21:40:58.828594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.828572    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f504460b-0099-4ecb-a8f8-e3493bae4258" containerName="istio-proxy"
Nov 03 21:40:58.832270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.832251    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.889776    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a3dc77-openshift-default-dockercfg-qtvdp\""
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.895928    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896030    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6m2\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896092    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896116    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896253    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896334    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896439    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.906683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.896485    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.940230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.940191    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:40:58.997461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997427    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6m2\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997466    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997486    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997525    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997605    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997685    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997741    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997793    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997925    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.997978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.997961    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.998175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.998152    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.998262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.998198    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.998320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.998303    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:58.998912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:58.998884    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.000772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.000749    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.000914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.000806    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.055350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.055321    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.056534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.056513    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6m2\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\") pod \"e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") " pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.146291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.146212    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:40:59.580560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.580491    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:40:59.581481 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:40:59.581450    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d352da3_493f_4470_88a7_99fba4032d7c.slice/crio-8ad12a195e752a07e95261e284dd8c07091583791ef3bdc26085a0cc671d0102 WatchSource:0}: Error finding container 8ad12a195e752a07e95261e284dd8c07091583791ef3bdc26085a0cc671d0102: Status 404 returned error can't find the container with id 8ad12a195e752a07e95261e284dd8c07091583791ef3bdc26085a0cc671d0102
Nov 03 21:40:59.583998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.583966    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:59.584095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.584029    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:59.584095 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.584058    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:40:59.954000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.953958    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerStarted","Data":"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"}
Nov 03 21:40:59.954441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:40:59.954018    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerStarted","Data":"8ad12a195e752a07e95261e284dd8c07091583791ef3bdc26085a0cc671d0102"}
Nov 03 21:41:00.070570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:00.070520    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podStartSLOduration=2.070507275 podStartE2EDuration="2.070507275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:00.069254541 +0000 UTC m=+3161.700568018" watchObservedRunningTime="2026-11-03 21:41:00.070507275 +0000 UTC m=+3161.701820818"
Nov 03 21:41:00.146326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:00.146296    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:41:00.152919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:00.152898    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:41:00.961740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:00.961696    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:41:00.962746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:00.962723    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:41:13.726994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.725404    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:41:13.730854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.730825    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:13.776089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.775736    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:41:13.817321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.817291    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a3dc77\""
Nov 03 21:41:13.826574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.826550    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a3dc77-dockercfg-krd79\""
Nov 03 21:41:13.863581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.863552    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:13.863721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.863586    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mbx\" (UniqueName: \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:13.964486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.964458    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:13.964616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.964503    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mbx\" (UniqueName: \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:13.965431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:13.965404    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:14.022415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:14.022355    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mbx\" (UniqueName: \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\") pod \"payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:14.047364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:14.047337    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:14.417907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:14.417877    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:41:14.419835 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:41:14.419803    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5b88d9_de6f_449e_a4ea_50db7739b1c7.slice/crio-2b225fdac44d0fd67277671ae35f41a115b08c4a9678f89dbb118aaf530c9e9b WatchSource:0}: Error finding container 2b225fdac44d0fd67277671ae35f41a115b08c4a9678f89dbb118aaf530c9e9b: Status 404 returned error can't find the container with id 2b225fdac44d0fd67277671ae35f41a115b08c4a9678f89dbb118aaf530c9e9b
Nov 03 21:41:14.847547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:14.847515    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:41:15.063459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:15.063422    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" event={"ID":"6e5b88d9-de6f-449e-a4ea-50db7739b1c7","Type":"ContainerStarted","Data":"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"}
Nov 03 21:41:15.063734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:15.063488    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" event={"ID":"6e5b88d9-de6f-449e-a4ea-50db7739b1c7","Type":"ContainerStarted","Data":"2b225fdac44d0fd67277671ae35f41a115b08c4a9678f89dbb118aaf530c9e9b"}
Nov 03 21:41:15.063734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:15.063519    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:15.172996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:15.172916    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" podStartSLOduration=2.172902024 podStartE2EDuration="2.172902024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:15.172697131 +0000 UTC m=+3176.804010609" watchObservedRunningTime="2026-11-03 21:41:15.172902024 +0000 UTC m=+3176.804215500"
Nov 03 21:41:19.262986 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:41:19.262937    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:41:21.227672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.227629    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"]
Nov 03 21:41:21.232622 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.232599    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.286681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.286655    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"]
Nov 03 21:41:21.329037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.329010    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:41:21.329136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.329105    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:41:21.353535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.353512    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8xc\" (UniqueName: \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.353673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.353583    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.353797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.353776    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.454785 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.454758    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8xc\" (UniqueName: \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.454890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.454822    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.454964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.454890    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.455650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.455616    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.457384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.457362    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.512815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.512758    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8xc\" (UniqueName: \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\") pod \"maas-api-revoke-sub-e0d43e84-lbsh6\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.545926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.545906    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:21.924380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:21.924259    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"]
Nov 03 21:41:22.122078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:22.122045    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6" event={"ID":"ff937b78-c65b-40af-b11a-5702356e9cd2","Type":"ContainerStarted","Data":"c7f3b47255b3868f2e96c87637cc3fdb0c060d845e8350262e2b645689d63e73"}
Nov 03 21:41:23.130239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:23.130160    5533 generic.go:356] "Generic (PLEG): container finished" podID="ff937b78-c65b-40af-b11a-5702356e9cd2" containerID="26f0074f5818e9fa9561142300096a3468a83a2cb25336480b6bade9b077f2e5" exitCode=0
Nov 03 21:41:23.130691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:23.130241    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6" event={"ID":"ff937b78-c65b-40af-b11a-5702356e9cd2","Type":"ContainerDied","Data":"26f0074f5818e9fa9561142300096a3468a83a2cb25336480b6bade9b077f2e5"}
Nov 03 21:41:24.296010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.295987    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:24.366047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.366015    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:41:24.389803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.389737    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\") pod \"ff937b78-c65b-40af-b11a-5702356e9cd2\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") "
Nov 03 21:41:24.389803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.389783    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\" (UniqueName: \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\") pod \"ff937b78-c65b-40af-b11a-5702356e9cd2\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") "
Nov 03 21:41:24.390019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.389880    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\") pod \"ff937b78-c65b-40af-b11a-5702356e9cd2\" (UID: \"ff937b78-c65b-40af-b11a-5702356e9cd2\") "
Nov 03 21:41:24.390279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.390255    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca" pod "ff937b78-c65b-40af-b11a-5702356e9cd2" (UID: "ff937b78-c65b-40af-b11a-5702356e9cd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:24.390510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.390456    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:24.391758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.391727    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca" pod "ff937b78-c65b-40af-b11a-5702356e9cd2" (UID: "ff937b78-c65b-40af-b11a-5702356e9cd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:41:24.391972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.391951    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc" pod "ff937b78-c65b-40af-b11a-5702356e9cd2" (UID: "ff937b78-c65b-40af-b11a-5702356e9cd2"). InnerVolumeSpecName "kube-api-access-ws8xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:24.497065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.494238    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937b78-c65b-40af-b11a-5702356e9cd2-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:24.497065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:24.494272    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8xc\" (UniqueName: \"kubernetes.io/projected/ff937b78-c65b-40af-b11a-5702356e9cd2-kube-api-access-ws8xc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:25.146019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:25.145992    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"
Nov 03 21:41:25.146197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:25.146024    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6" event={"ID":"ff937b78-c65b-40af-b11a-5702356e9cd2","Type":"ContainerDied","Data":"c7f3b47255b3868f2e96c87637cc3fdb0c060d845e8350262e2b645689d63e73"}
Nov 03 21:41:25.146197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:25.146080    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f3b47255b3868f2e96c87637cc3fdb0c060d845e8350262e2b645689d63e73"
Nov 03 21:41:26.081954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:26.081921    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:41:27.865468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:27.865431    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:41:27.865917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:27.865759    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" podUID="088d43cc-61c0-475f-81ad-dc85454a08a7" containerName="istio-proxy" containerID="cri-o://4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953" gracePeriod=30
Nov 03 21:41:31.932710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.932671    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"]
Nov 03 21:41:31.933358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.933340    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ff937b78-c65b-40af-b11a-5702356e9cd2" containerName="revoke-keys"
Nov 03 21:41:31.933411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.933361    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff937b78-c65b-40af-b11a-5702356e9cd2" containerName="revoke-keys"
Nov 03 21:41:31.933474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.933465    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff937b78-c65b-40af-b11a-5702356e9cd2" containerName="revoke-keys"
Nov 03 21:41:31.936996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.936978    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:31.975343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.975316    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8tm6\" (UniqueName: \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:31.975482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.975373    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:31.975574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.975552    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:31.996214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:31.996182    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"]
Nov 03 21:41:32.033401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.033367    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:41:32.033537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.033421    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:41:32.076959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.076927    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8tm6\" (UniqueName: \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.077126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.076987    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.077126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.077073    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.077663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.077623    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.079331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.079311    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.134426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.134401    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8tm6\" (UniqueName: \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\") pod \"maas-api-revoke-sub-1208cb28-f5rtc\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.251141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.251073    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:32.627231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:32.627200    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"]
Nov 03 21:41:32.628292 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:41:32.628269    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702c15e1_e9ec_4683_abb7_8b9ef2cf1bff.slice/crio-342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1 WatchSource:0}: Error finding container 342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1: Status 404 returned error can't find the container with id 342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1
Nov 03 21:41:33.128457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.128408    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:41:33.188984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.188955    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.188998    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189039    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189084    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189130    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189231    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189261    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189293    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189351    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\") pod \"088d43cc-61c0-475f-81ad-dc85454a08a7\" (UID: \"088d43cc-61c0-475f-81ad-dc85454a08a7\") "
Nov 03 21:41:33.189535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189501    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189815    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189837    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.189876    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.190032    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.190051    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088d43cc-61c0-475f-81ad-dc85454a08a7-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.190066    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.190080    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.190136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.190131    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:33.191408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.191383    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:33.191895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.191867    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:33.192046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.192023    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:33.192516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.192492    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f" pod "088d43cc-61c0-475f-81ad-dc85454a08a7" (UID: "088d43cc-61c0-475f-81ad-dc85454a08a7"). InnerVolumeSpecName "kube-api-access-ndm2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:33.206171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.206146    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc" event={"ID":"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff","Type":"ContainerStarted","Data":"76ba667b6ebee30e237b60f07dd3b16ad52e5086937b4f65c4cca9da397629e8"}
Nov 03 21:41:33.206277 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.206192    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc" event={"ID":"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff","Type":"ContainerStarted","Data":"342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1"}
Nov 03 21:41:33.208673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.208650    5533 generic.go:356] "Generic (PLEG): container finished" podID="088d43cc-61c0-475f-81ad-dc85454a08a7" containerID="4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953" exitCode=0
Nov 03 21:41:33.208782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.208684    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"
Nov 03 21:41:33.208782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.208730    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" event={"ID":"088d43cc-61c0-475f-81ad-dc85454a08a7","Type":"ContainerDied","Data":"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"}
Nov 03 21:41:33.208919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.208790    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf" event={"ID":"088d43cc-61c0-475f-81ad-dc85454a08a7","Type":"ContainerDied","Data":"441e3e348929376a5f3ab1f775f71205eed0024adc842212fca70a952de08904"}
Nov 03 21:41:33.208919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.208823    5533 scope.go:122] "RemoveContainer" containerID="4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"
Nov 03 21:41:33.220628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.220611    5533 scope.go:122] "RemoveContainer" containerID="4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"
Nov 03 21:41:33.220913 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:41:33.220895    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953\": container with ID starting with 4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953 not found: ID does not exist" containerID="4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"
Nov 03 21:41:33.220961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.220921    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953"} err="failed to get container status \"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953\": rpc error: code = NotFound desc = could not find container \"4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953\": container with ID starting with 4ad5a635f94755f3f21014af48cc14919a55aa6783e858d1d29f9d468ae96953 not found: ID does not exist"
Nov 03 21:41:33.292176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.291924    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.292176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.291953    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndm2f\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-kube-api-access-ndm2f\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.292176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.291968    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.292176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.291981    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.292176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.291995    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088d43cc-61c0-475f-81ad-dc85454a08a7-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:33.401231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.401135    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc" podStartSLOduration=2.401122379 podStartE2EDuration="2.401122379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:33.40024182 +0000 UTC m=+3195.031555299" watchObservedRunningTime="2026-11-03 21:41:33.401122379 +0000 UTC m=+3195.032435856"
Nov 03 21:41:33.508272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.508242    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:41:33.512228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:33.512203    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-21475a-openshift-default-678c5f894f-pzpwf"]
Nov 03 21:41:34.218064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:34.218035    5533 generic.go:356] "Generic (PLEG): container finished" podID="702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" containerID="76ba667b6ebee30e237b60f07dd3b16ad52e5086937b4f65c4cca9da397629e8" exitCode=0
Nov 03 21:41:34.218460 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:34.218081    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc" event={"ID":"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff","Type":"ContainerDied","Data":"76ba667b6ebee30e237b60f07dd3b16ad52e5086937b4f65c4cca9da397629e8"}
Nov 03 21:41:34.498171 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:34.498144    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088d43cc-61c0-475f-81ad-dc85454a08a7" path="/var/lib/kubelet/pods/088d43cc-61c0-475f-81ad-dc85454a08a7/volumes"
Nov 03 21:41:35.384397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.384370    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:35.516570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.516493    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\" (UniqueName: \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\") pod \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") "
Nov 03 21:41:35.516758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.516593    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\") pod \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") "
Nov 03 21:41:35.516758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.516626    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\") pod \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\" (UID: \"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff\") "
Nov 03 21:41:35.517006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.516980    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca" pod "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" (UID: "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:35.519329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.519300    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca" pod "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" (UID: "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:41:35.519436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.519417    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6" pod "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" (UID: "702c15e1-e9ec-4683-abb7-8b9ef2cf1bff"). InnerVolumeSpecName "kube-api-access-t8tm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:35.618178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.618010    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8tm6\" (UniqueName: \"kubernetes.io/projected/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-kube-api-access-t8tm6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:35.618178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.618037    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:35.618178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:35.618051    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:36.236771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:36.236737    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"
Nov 03 21:41:36.236953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:36.236738    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc" event={"ID":"702c15e1-e9ec-4683-abb7-8b9ef2cf1bff","Type":"ContainerDied","Data":"342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1"}
Nov 03 21:41:36.236953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:36.236863    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342ec62585e0e81ba13d87a483ba2f885d93847ce69f609e2bd5d70f1a3263c1"
Nov 03 21:41:39.386128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.386086    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" podUID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" containerName="main" containerID="cri-o://e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660" gracePeriod=45
Nov 03 21:41:39.652387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.652329    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:41:39.761757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.761720    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.761984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.761800    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\" (UniqueName: \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.761984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.761981    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.762148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762017    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.762148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762019    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:39.762148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762060    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.762148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762091    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\" (UniqueName: \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\") pod \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\" (UID: \"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae\") "
Nov 03 21:41:39.762383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762326    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:39.762630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762608    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:39.762630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762648    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:39.762865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.762841    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:39.764857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.764827    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:39.766120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.765182    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:41:39.767179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.767156    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m" pod "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" (UID: "efe8536c-0be4-4dad-aed8-9c8dc2b8ceae"). InnerVolumeSpecName "kube-api-access-skm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:39.864449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.864414    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skm9m\" (UniqueName: \"kubernetes.io/projected/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-kube-api-access-skm9m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:39.864585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.864452    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:39.864585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.864465    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:39.864585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:39.864479    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:41:40.270659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.269861    5533 generic.go:356] "Generic (PLEG): container finished" podID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" containerID="e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660" exitCode=0
Nov 03 21:41:40.270659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.269925    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"
Nov 03 21:41:40.270659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.269944    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" event={"ID":"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae","Type":"ContainerDied","Data":"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"}
Nov 03 21:41:40.270659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.269997    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58" event={"ID":"efe8536c-0be4-4dad-aed8-9c8dc2b8ceae","Type":"ContainerDied","Data":"760f4f2fd3f5d1e494c73fc59d688e46f43a54ef48468e650479921b4a95d87c"}
Nov 03 21:41:40.270659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.270020    5533 scope.go:122] "RemoveContainer" containerID="e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"
Nov 03 21:41:40.284419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.284400    5533 scope.go:122] "RemoveContainer" containerID="e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"
Nov 03 21:41:40.284780 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:41:40.284753    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660\": container with ID starting with e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660 not found: ID does not exist" containerID="e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"
Nov 03 21:41:40.284880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.284790    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660"} err="failed to get container status \"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660\": rpc error: code = NotFound desc = could not find container \"e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660\": container with ID starting with e01e122300407fee44efc2febf5339b76ce24678d69af044fecd1171e35ed660 not found: ID does not exist"
Nov 03 21:41:40.426248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.426213    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:41:40.473187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.473155    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6219b/auth-test-model-a6219b-kserve-9d687fc-8fr58"]
Nov 03 21:41:40.495402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:40.495369    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" path="/var/lib/kubelet/pods/efe8536c-0be4-4dad-aed8-9c8dc2b8ceae/volumes"
Nov 03 21:41:49.282365 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:41:49.282326    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:41:58.429136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.429100    5533 generic.go:356] "Generic (PLEG): container finished" podID="168a1c45-db66-4eb3-8327-978472d5538d" containerID="ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc" exitCode=0
Nov 03 21:41:58.429604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.429188    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerDied","Data":"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"}
Nov 03 21:41:58.429668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.429616    5533 scope.go:122] "RemoveContainer" containerID="ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"
Nov 03 21:41:58.431245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.431217    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:58.431342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.431278    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:58.431342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.431311    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:41:58.474403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:58.474383    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:41:59.438194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:59.438156    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerStarted","Data":"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"}
Nov 03 21:41:59.579995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:59.579957    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:41:59.582865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:59.582770    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 21:41:59.583036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:41:59.582907    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 21:42:00.576032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:00.575988    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 21:42:00.576516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:00.576072    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 21:42:01.577015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:01.576982    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 21:42:01.577592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:01.577562    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 21:42:02.577992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:02.577953    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 21:42:02.578570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:02.578539    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 21:42:03.574967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:03.574929    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 21:42:03.575279 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:03.575247    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 21:42:04.576818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:04.576780    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:42:04.577319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:04.577298    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:42:04.577997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:04.577975    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:42:08.934682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:08.934649    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:42:13.177890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.177854    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:42:13.178650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178610    5533 cpu_manager.go:414] "containerMap: removing container" podUID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" containerName="main"
Nov 03 21:42:13.178650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178647    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" containerName="main"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178663    5533 cpu_manager.go:414] "containerMap: removing container" podUID="088d43cc-61c0-475f-81ad-dc85454a08a7" containerName="istio-proxy"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178672    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088d43cc-61c0-475f-81ad-dc85454a08a7" containerName="istio-proxy"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178703    5533 cpu_manager.go:414] "containerMap: removing container" podUID="702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" containerName="revoke-keys"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178710    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" containerName="revoke-keys"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178816    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" containerName="revoke-keys"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178828    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe8536c-0be4-4dad-aed8-9c8dc2b8ceae" containerName="main"
Nov 03 21:42:13.178834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.178833    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="088d43cc-61c0-475f-81ad-dc85454a08a7" containerName="istio-proxy"
Nov 03 21:42:13.183339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.183314    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.232866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.232842    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2057ca-openshift-default-dockercfg-6rl6w\""
Nov 03 21:42:13.245587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245558    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnq7\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.245758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245606    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.245758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245686    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.245758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245718    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.245949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245871    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.245949 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245911    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.246070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.245979    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.246130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.246093    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.246187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.246146    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.282888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.282862    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:42:13.347856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.347129    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.348571 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.348546    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.349461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.349440    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.350668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.350630    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.351151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.351133    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.351765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.351076    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.352231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.349371    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.352386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.351511    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.352525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.348470    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.352699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.352681    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.352926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.352910    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.353355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.353337    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.353866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.353846    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnq7\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.354349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.353267    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.357249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.357227    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.397686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.390231    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.410134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.410104    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.411413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.411379    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnq7\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\") pod \"e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") " pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.497034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.496969    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:13.928951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.928924    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:42:13.930105 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:13.930073    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e9124_d79b_4ae9_99e9_1247d686b96b.slice/crio-2a399c16a2dc5d86b2068fbf2c19c13e60404230e471d9692e7513d0d3c37432 WatchSource:0}: Error finding container 2a399c16a2dc5d86b2068fbf2c19c13e60404230e471d9692e7513d0d3c37432: Status 404 returned error can't find the container with id 2a399c16a2dc5d86b2068fbf2c19c13e60404230e471d9692e7513d0d3c37432
Nov 03 21:42:13.932723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.932689    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:13.932835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.932756    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:13.932835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:13.932786    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:14.566407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:14.566361    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerStarted","Data":"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"}
Nov 03 21:42:14.566924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:14.566426    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerStarted","Data":"2a399c16a2dc5d86b2068fbf2c19c13e60404230e471d9692e7513d0d3c37432"}
Nov 03 21:42:14.683400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:14.683350    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podStartSLOduration=1.6833379160000002 podStartE2EDuration="1.683337916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:14.679593888 +0000 UTC m=+3236.310907367" watchObservedRunningTime="2026-11-03 21:42:14.683337916 +0000 UTC m=+3236.314651393"
Nov 03 21:42:15.497175 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:15.497137    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:15.503609 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:15.503584    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:15.574616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:15.574592    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:15.575458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:15.575437    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:42:19.261574 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:42:19.261535    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:42:19.775006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:19.774973    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:42:31.131593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.131558    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:42:31.137352 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.137331    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.173150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.173121    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rk89\" (UniqueName: \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.173272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.173239    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.194682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.194655    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:42:31.236237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.236208    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2057ca\""
Nov 03 21:42:31.236782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.236759    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2057ca-dockercfg-mvswh\""
Nov 03 21:42:31.291658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.282301    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.291658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.282476    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rk89\" (UniqueName: \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.291658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.285128    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.305678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.302289    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:42:31.308927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.308905    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.340603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.340580    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rk89\" (UniqueName: \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\") pod \"payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.350012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.349985    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:42:31.383291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.383220    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgpp\" (UniqueName: \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.383429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.383302    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.451255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.451225    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:31.484108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.484081    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgpp\" (UniqueName: \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.484249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.484184    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.484965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.484944    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.542408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.542377    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgpp\" (UniqueName: \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\") pod \"payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.623306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.623276    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:31.830862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:31.830835    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:42:31.830985 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:31.830912    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343ab693_fc33_41fe_8c73_32c72c033860.slice/crio-48e2fe55ffb432f62f30d396aa9fc42b1bd32d656e912119b622b355576c503d WatchSource:0}: Error finding container 48e2fe55ffb432f62f30d396aa9fc42b1bd32d656e912119b622b355576c503d: Status 404 returned error can't find the container with id 48e2fe55ffb432f62f30d396aa9fc42b1bd32d656e912119b622b355576c503d
Nov 03 21:42:32.012487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.012458    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:42:32.013569 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:32.013545    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7102a4_c470_4446_8941_bf0eb614d0f2.slice/crio-367be500a484cc827ad3477099d49f1845475e5e58264307a8defe100dd118ca WatchSource:0}: Error finding container 367be500a484cc827ad3477099d49f1845475e5e58264307a8defe100dd118ca: Status 404 returned error can't find the container with id 367be500a484cc827ad3477099d49f1845475e5e58264307a8defe100dd118ca
Nov 03 21:42:32.534520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.534485    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:42:32.718446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.718391    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" event={"ID":"fd7102a4-c470-4446-8941-bf0eb614d0f2","Type":"ContainerStarted","Data":"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"}
Nov 03 21:42:32.718446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.718451    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" event={"ID":"fd7102a4-c470-4446-8941-bf0eb614d0f2","Type":"ContainerStarted","Data":"367be500a484cc827ad3477099d49f1845475e5e58264307a8defe100dd118ca"}
Nov 03 21:42:32.718714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.718493    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:32.720255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.720219    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" event={"ID":"343ab693-fc33-41fe-8c73-32c72c033860","Type":"ContainerStarted","Data":"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"}
Nov 03 21:42:32.720388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.720267    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" event={"ID":"343ab693-fc33-41fe-8c73-32c72c033860","Type":"ContainerStarted","Data":"48e2fe55ffb432f62f30d396aa9fc42b1bd32d656e912119b622b355576c503d"}
Nov 03 21:42:32.720388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.720291    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:32.833800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.831860    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" podStartSLOduration=1.831842099 podStartE2EDuration="1.831842099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:32.831150025 +0000 UTC m=+3254.462463503" watchObservedRunningTime="2026-11-03 21:42:32.831842099 +0000 UTC m=+3254.463155576"
Nov 03 21:42:32.940699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:32.940619    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" podStartSLOduration=1.940606171 podStartE2EDuration="1.940606171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:32.940526075 +0000 UTC m=+3254.571839553" watchObservedRunningTime="2026-11-03 21:42:32.940606171 +0000 UTC m=+3254.571919647"
Nov 03 21:42:34.795841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:34.795724    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" podUID="39ad35a0-76fa-4103-9118-f7acf9602c9f" containerName="main" containerID="cri-o://978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044" gracePeriod=45
Nov 03 21:42:35.067894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.067871    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:42:35.155877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.155844    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.155943    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.155986    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156054    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156082    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\" (UniqueName: \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156128    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\") pod \"39ad35a0-76fa-4103-9118-f7acf9602c9f\" (UID: \"39ad35a0-76fa-4103-9118-f7acf9602c9f\") "
Nov 03 21:42:35.156331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156303    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:35.156396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156329    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:35.156464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156438    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:35.156718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156699    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.156807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156722    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.156807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.156737    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.158657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.158605    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "kube-api-access-nlc9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:35.158790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.158673    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:35.158790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.158688    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs" pod "39ad35a0-76fa-4103-9118-f7acf9602c9f" (UID: "39ad35a0-76fa-4103-9118-f7acf9602c9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:35.257729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.257702    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ad35a0-76fa-4103-9118-f7acf9602c9f-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.257729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.257727    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ad35a0-76fa-4103-9118-f7acf9602c9f-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.257895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.257741    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlc9n\" (UniqueName: \"kubernetes.io/projected/39ad35a0-76fa-4103-9118-f7acf9602c9f-kube-api-access-nlc9n\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:35.775247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.775211    5533 generic.go:356] "Generic (PLEG): container finished" podID="39ad35a0-76fa-4103-9118-f7acf9602c9f" containerID="978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044" exitCode=0
Nov 03 21:42:35.775401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.775250    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"
Nov 03 21:42:35.775401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.775289    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" event={"ID":"39ad35a0-76fa-4103-9118-f7acf9602c9f","Type":"ContainerDied","Data":"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"}
Nov 03 21:42:35.775401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.775335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk" event={"ID":"39ad35a0-76fa-4103-9118-f7acf9602c9f","Type":"ContainerDied","Data":"d172a470f5b27bbf7b5f5500f76eb651b82073e91af908d6869ff40c12a99504"}
Nov 03 21:42:35.775401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.775351    5533 scope.go:122] "RemoveContainer" containerID="978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"
Nov 03 21:42:35.787887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.787857    5533 scope.go:122] "RemoveContainer" containerID="978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"
Nov 03 21:42:35.788175 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:42:35.788156    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044\": container with ID starting with 978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044 not found: ID does not exist" containerID="978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"
Nov 03 21:42:35.788265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.788184    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044"} err="failed to get container status \"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044\": rpc error: code = NotFound desc = could not find container \"978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044\": container with ID starting with 978bc6104aeefeeeb68ec909f685cd08c10bc583aa2d9835bf5ac1c7cfe10044 not found: ID does not exist"
Nov 03 21:42:35.932220 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.932192    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:42:35.978773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:35.978747    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/auth-test-model-c11538-kserve-7c5677fc4f-dd2pk"]
Nov 03 21:42:36.499592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:36.499546    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ad35a0-76fa-4103-9118-f7acf9602c9f" path="/var/lib/kubelet/pods/39ad35a0-76fa-4103-9118-f7acf9602c9f/volumes"
Nov 03 21:42:42.660776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.660740    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"]
Nov 03 21:42:42.661705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.661682    5533 cpu_manager.go:414] "containerMap: removing container" podUID="39ad35a0-76fa-4103-9118-f7acf9602c9f" containerName="main"
Nov 03 21:42:42.661705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.661706    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ad35a0-76fa-4103-9118-f7acf9602c9f" containerName="main"
Nov 03 21:42:42.661888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.661880    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ad35a0-76fa-4103-9118-f7acf9602c9f" containerName="main"
Nov 03 21:42:42.665916 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.665893    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.719161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.719130    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"]
Nov 03 21:42:42.752370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.752335    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.752663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.752623    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.752856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.752835    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5ws\" (UniqueName: \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.761416 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.761381    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:42.761574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.761445    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:42:42.854544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.854512    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.854544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.854547    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5ws\" (UniqueName: \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.854833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.854583    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.855340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.855316    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.857104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.857074    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.912545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.912483    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5ws\" (UniqueName: \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\") pod \"maas-api-revoke-sub-0ce0a736-smdqm\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:42.980713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:42.980678    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:43.369109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:43.367727    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"]
Nov 03 21:42:43.369109 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:43.368615    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae267d51_0c42_42e1_872a_b8e13fcc0c64.slice/crio-6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7 WatchSource:0}: Error finding container 6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7: Status 404 returned error can't find the container with id 6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7
Nov 03 21:42:43.729797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:43.729770    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:42:43.730194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:43.729837    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:42:43.846781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:43.846749    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm" event={"ID":"ae267d51-0c42-42e1-872a-b8e13fcc0c64","Type":"ContainerStarted","Data":"f67edd6e3f73bef27a880e7a80111f1fc5e3d574ae034e3ee7c34ad2676f24e7"}
Nov 03 21:42:43.846953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:43.846804    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm" event={"ID":"ae267d51-0c42-42e1-872a-b8e13fcc0c64","Type":"ContainerStarted","Data":"6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7"}
Nov 03 21:42:44.058237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:44.058119    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm" podStartSLOduration=2.058101559 podStartE2EDuration="2.058101559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:44.056873671 +0000 UTC m=+3265.688187151" watchObservedRunningTime="2026-11-03 21:42:44.058101559 +0000 UTC m=+3265.689415038"
Nov 03 21:42:44.858594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:44.858535    5533 generic.go:356] "Generic (PLEG): container finished" podID="ae267d51-0c42-42e1-872a-b8e13fcc0c64" containerID="f67edd6e3f73bef27a880e7a80111f1fc5e3d574ae034e3ee7c34ad2676f24e7" exitCode=0
Nov 03 21:42:44.859111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:44.858616    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm" event={"ID":"ae267d51-0c42-42e1-872a-b8e13fcc0c64","Type":"ContainerDied","Data":"f67edd6e3f73bef27a880e7a80111f1fc5e3d574ae034e3ee7c34ad2676f24e7"}
Nov 03 21:42:45.868280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.868195    5533 generic.go:356] "Generic (PLEG): container finished" podID="4d352da3-493f-4470-88a7-99fba4032d7c" containerID="7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed" exitCode=0
Nov 03 21:42:45.868817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.868274    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerDied","Data":"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"}
Nov 03 21:42:45.868817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.868630    5533 scope.go:122] "RemoveContainer" containerID="7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"
Nov 03 21:42:45.870248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.870219    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:45.870368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.870288    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:45.870368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.870322    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:42:45.962863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:45.962369    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:42:46.025853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.025821    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:46.105474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.105444    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\" (UniqueName: \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\") pod \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") "
Nov 03 21:42:46.105688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.105583    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\") pod \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") "
Nov 03 21:42:46.105757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.105734    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\") pod \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\" (UID: \"ae267d51-0c42-42e1-872a-b8e13fcc0c64\") "
Nov 03 21:42:46.105964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.105931    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca" pod "ae267d51-0c42-42e1-872a-b8e13fcc0c64" (UID: "ae267d51-0c42-42e1-872a-b8e13fcc0c64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:46.106150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.106132    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:46.107686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.107662    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws" pod "ae267d51-0c42-42e1-872a-b8e13fcc0c64" (UID: "ae267d51-0c42-42e1-872a-b8e13fcc0c64"). InnerVolumeSpecName "kube-api-access-mc5ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:46.107844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.107828    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca" pod "ae267d51-0c42-42e1-872a-b8e13fcc0c64" (UID: "ae267d51-0c42-42e1-872a-b8e13fcc0c64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:46.207677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.207622    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5ws\" (UniqueName: \"kubernetes.io/projected/ae267d51-0c42-42e1-872a-b8e13fcc0c64-kube-api-access-mc5ws\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:46.207677 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.207672    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae267d51-0c42-42e1-872a-b8e13fcc0c64-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:46.890417 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.890330    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerStarted","Data":"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"}
Nov 03 21:42:46.892250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.892225    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"
Nov 03 21:42:46.892403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.892384    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm" event={"ID":"ae267d51-0c42-42e1-872a-b8e13fcc0c64","Type":"ContainerDied","Data":"6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7"}
Nov 03 21:42:46.892499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:46.892490    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a5869923a776564d9e9077e6aadd8fc1de7ecdd0700dec352f9d0d36b2928c7"
Nov 03 21:42:47.146878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.146801    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:42:47.148063 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.148032    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:47.148219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.148089    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:47.561658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.560856    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" podUID="30df6e44-f914-4747-af20-62e15d22c13d" containerName="main" containerID="cri-o://f2ffc4ea2c1bd0fa32aaf2f8227cbe05032886d8d76b8760a76166c1ce55e608" gracePeriod=45
Nov 03 21:42:47.936533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.936505    5533 generic.go:356] "Generic (PLEG): container finished" podID="30df6e44-f914-4747-af20-62e15d22c13d" containerID="f2ffc4ea2c1bd0fa32aaf2f8227cbe05032886d8d76b8760a76166c1ce55e608" exitCode=0
Nov 03 21:42:47.937062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.937037    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" event={"ID":"30df6e44-f914-4747-af20-62e15d22c13d","Type":"ContainerDied","Data":"f2ffc4ea2c1bd0fa32aaf2f8227cbe05032886d8d76b8760a76166c1ce55e608"}
Nov 03 21:42:47.957738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:47.951691    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:42:48.036579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036552    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.036772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036610    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\" (UniqueName: \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.036772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036643    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.036772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036690    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.036772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036708    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.036772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.036747    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\" (UniqueName: \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\") pod \"30df6e44-f914-4747-af20-62e15d22c13d\" (UID: \"30df6e44-f914-4747-af20-62e15d22c13d\") "
Nov 03 21:42:48.037078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.037007    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:48.037078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.037020    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:48.037234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.037200    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.037234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.037232    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.037393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.037373    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:48.038977 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.038951    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:48.039101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.039020    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:48.039156 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.039102    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2" pod "30df6e44-f914-4747-af20-62e15d22c13d" (UID: "30df6e44-f914-4747-af20-62e15d22c13d"). InnerVolumeSpecName "kube-api-access-c8pw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:48.137922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.137888    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8pw2\" (UniqueName: \"kubernetes.io/projected/30df6e44-f914-4747-af20-62e15d22c13d-kube-api-access-c8pw2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.137922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.137916    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.137922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.137927    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30df6e44-f914-4747-af20-62e15d22c13d-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.138165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.137936    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30df6e44-f914-4747-af20-62e15d22c13d-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:48.146717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.146691    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:48.146866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.146737    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:48.200310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.200283    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:42:48.201376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.201347    5533 cpu_manager.go:414] "containerMap: removing container" podUID="30df6e44-f914-4747-af20-62e15d22c13d" containerName="main"
Nov 03 21:42:48.201547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.201533    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30df6e44-f914-4747-af20-62e15d22c13d" containerName="main"
Nov 03 21:42:48.201690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.201677    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ae267d51-0c42-42e1-872a-b8e13fcc0c64" containerName="revoke-keys"
Nov 03 21:42:48.201786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.201774    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae267d51-0c42-42e1-872a-b8e13fcc0c64" containerName="revoke-keys"
Nov 03 21:42:48.202071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.202058    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="30df6e44-f914-4747-af20-62e15d22c13d" containerName="main"
Nov 03 21:42:48.202166 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.202157    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae267d51-0c42-42e1-872a-b8e13fcc0c64" containerName="revoke-keys"
Nov 03 21:42:48.206393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.206371    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.238939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.238913    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.239084 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.239062    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6km8c\" (UniqueName: \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.271665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.269819    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d5e2db-dockercfg-brtwh\""
Nov 03 21:42:48.271665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.270485    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d5e2db\""
Nov 03 21:42:48.296278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.296245    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:42:48.364667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.352453    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.365133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.365097    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6km8c\" (UniqueName: \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.402661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.366735    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.420659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.419723    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:42:48.428705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.425169    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.429322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.429292    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:42:48.435302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.435277    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6km8c\" (UniqueName: \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\") pod \"payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.519926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.519896    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:48.584661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.575587    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccwh\" (UniqueName: \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.585122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.585092    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.687656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.686319    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.689400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.689375    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccwh\" (UniqueName: \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.689976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.689240    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.747587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.747561    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccwh\" (UniqueName: \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\") pod \"payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:48.932702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.932673    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:42:48.933305 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:48.933279    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147306bc_00da_47c2_9e2f_ca6b566485f4.slice/crio-0dcf7363481cb3afd89e5aa62ce9730229600d1573dcc49b55afe87c17d86efc WatchSource:0}: Error finding container 0dcf7363481cb3afd89e5aa62ce9730229600d1573dcc49b55afe87c17d86efc: Status 404 returned error can't find the container with id 0dcf7363481cb3afd89e5aa62ce9730229600d1573dcc49b55afe87c17d86efc
Nov 03 21:42:48.945580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.945555    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" event={"ID":"147306bc-00da-47c2-9e2f-ca6b566485f4","Type":"ContainerStarted","Data":"0dcf7363481cb3afd89e5aa62ce9730229600d1573dcc49b55afe87c17d86efc"}
Nov 03 21:42:48.947019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.947000    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"
Nov 03 21:42:48.947113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.947037    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h" event={"ID":"30df6e44-f914-4747-af20-62e15d22c13d","Type":"ContainerDied","Data":"7534bf5c08f4c5735eba3b72a461f11e565e5ddd29e180610c9b43b1de54326f"}
Nov 03 21:42:48.947113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:48.947094    5533 scope.go:122] "RemoveContainer" containerID="f2ffc4ea2c1bd0fa32aaf2f8227cbe05032886d8d76b8760a76166c1ce55e608"
Nov 03 21:42:49.044805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.044777    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:49.107249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.107215    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:42:49.111197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.111169    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/auth-test-model-f0dc66-kserve-7f74c7497c-q687h"]
Nov 03 21:42:49.147958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.147934    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:49.148130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.148105    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:49.288062 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:42:49.287975    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:42:49.464658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.450837    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:42:49.979482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.979444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" event={"ID":"147306bc-00da-47c2-9e2f-ca6b566485f4","Type":"ContainerStarted","Data":"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"}
Nov 03 21:42:49.980032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.979660    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:42:49.982324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.982297    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" event={"ID":"9c649afa-7c53-4c91-a325-72260ba6484c","Type":"ContainerStarted","Data":"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"}
Nov 03 21:42:49.982454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.982343    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" event={"ID":"9c649afa-7c53-4c91-a325-72260ba6484c","Type":"ContainerStarted","Data":"1dada78fd989489b4f8963bd56f4dc98ec461f61db25192f52b8d5cf3f2b1815"}
Nov 03 21:42:49.982454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:49.982424    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:42:50.091850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.091804    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" podStartSLOduration=2.091790288 podStartE2EDuration="2.091790288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:50.089272551 +0000 UTC m=+3271.720586027" watchObservedRunningTime="2026-11-03 21:42:50.091790288 +0000 UTC m=+3271.723103765"
Nov 03 21:42:50.146558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.146528    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:50.146707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.146582    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:50.202369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.202318    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" podStartSLOduration=2.202304034 podStartE2EDuration="2.202304034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:50.200034676 +0000 UTC m=+3271.831348152" watchObservedRunningTime="2026-11-03 21:42:50.202304034 +0000 UTC m=+3271.833617516"
Nov 03 21:42:50.507134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.507098    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30df6e44-f914-4747-af20-62e15d22c13d" path="/var/lib/kubelet/pods/30df6e44-f914-4747-af20-62e15d22c13d/volumes"
Nov 03 21:42:50.936544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.936510    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:42:50.940236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.940218    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:50.995184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.995152    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c0fc5f-dockercfg-8nsvl\""
Nov 03 21:42:50.999281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:50.999250    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:42:51.040552 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.040527    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c0fc5f\""
Nov 03 21:42:51.122278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.122246    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vq7f\" (UniqueName: \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.122472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.122447    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.134348 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.134318    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:42:51.138470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.138453    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.146601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.146581    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:51.146709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.146614    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:51.201810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.201738    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:42:51.223714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.223687    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.223882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.223864    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq7f\" (UniqueName: \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.224377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.224359    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.283419 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.283383    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq7f\" (UniqueName: \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\") pod \"payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.324753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.324725    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4tm\" (UniqueName: \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.324878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.324858    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.425872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.425838    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4tm\" (UniqueName: \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.426061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.425937    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.426763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.426741    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.483617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.483545    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4tm\" (UniqueName: \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\") pod \"payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.574659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.573841    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:51.774709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.774558    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:51.950457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.950424    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:42:51.998932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:51.998904    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" event={"ID":"4e215a7c-aba4-401b-a776-ac2b30561e1c","Type":"ContainerStarted","Data":"ed3be5a43516161eab81d1b797eabb66e0981912d457a00b1697ba6493082ad2"}
Nov 03 21:42:52.146691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:52.146630    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 21:42:52.146920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:52.146716    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 21:42:52.150119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:52.150087    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:42:52.150508 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:42:52.150486    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a1ad7a_81eb_4c32_acb7_2f332900be1a.slice/crio-a71e9c69ffdbb52a04f1133c33a8a3d090838aa6277998a6bd9bf0d80641aa78 WatchSource:0}: Error finding container a71e9c69ffdbb52a04f1133c33a8a3d090838aa6277998a6bd9bf0d80641aa78: Status 404 returned error can't find the container with id a71e9c69ffdbb52a04f1133c33a8a3d090838aa6277998a6bd9bf0d80641aa78
Nov 03 21:42:53.010594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.010549    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" event={"ID":"98a1ad7a-81eb-4c32-acb7-2f332900be1a","Type":"ContainerStarted","Data":"f215bf1dcfd95f2cf6534bb372da0e56cc4a14f382266f0a663904f892b32f1f"}
Nov 03 21:42:53.011086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.010615    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" event={"ID":"98a1ad7a-81eb-4c32-acb7-2f332900be1a","Type":"ContainerStarted","Data":"a71e9c69ffdbb52a04f1133c33a8a3d090838aa6277998a6bd9bf0d80641aa78"}
Nov 03 21:42:53.011086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.010675    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:42:53.012409 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.012382    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" event={"ID":"4e215a7c-aba4-401b-a776-ac2b30561e1c","Type":"ContainerStarted","Data":"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"}
Nov 03 21:42:53.012518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.012441    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:42:53.122766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.122707    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" podStartSLOduration=2.122691318 podStartE2EDuration="2.122691318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:53.119826956 +0000 UTC m=+3274.751140434" watchObservedRunningTime="2026-11-03 21:42:53.122691318 +0000 UTC m=+3274.754004799"
Nov 03 21:42:53.152452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.152425    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:42:53.152771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.152752    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:42:53.153473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.153456    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:42:53.230463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.230414    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" podStartSLOduration=3.230402203 podStartE2EDuration="3.230402203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:53.228876009 +0000 UTC m=+3274.860189486" watchObservedRunningTime="2026-11-03 21:42:53.230402203 +0000 UTC m=+3274.861715680"
Nov 03 21:42:53.393295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.393223    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"]
Nov 03 21:42:53.397924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.397901    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.447328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.447300    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:42:53.447624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.447603    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:42:53.459657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.459619    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"]
Nov 03 21:42:53.558631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.558596    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bvzn\" (UniqueName: \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.558772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.558749    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.558927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.558910    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.659888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.659831    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.660009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.659994    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.660052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.660033    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bvzn\" (UniqueName: \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.660787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.660762    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.662167 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.662146    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:53.717958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:53.717937    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bvzn\" (UniqueName: \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\") pod \"maas-api-revoke-sub-49b68457-j4mjh\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:54.015796 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:54.015769    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:54.390696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:54.386922    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"]
Nov 03 21:42:55.027685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:55.027606    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh" event={"ID":"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf","Type":"ContainerStarted","Data":"71a8ca671d926d000740526fbd7bf4ec1ff195a7ed82c07ab447c363054edfe9"}
Nov 03 21:42:55.028098 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:55.027687    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh" event={"ID":"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf","Type":"ContainerStarted","Data":"6026da1d608a90fac868808008aa3affabe63abf1da2542797b46c9d1c339865"}
Nov 03 21:42:55.138238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:55.138193    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh" podStartSLOduration=2.138180385 podStartE2EDuration="2.138180385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:55.136825765 +0000 UTC m=+3276.768139241" watchObservedRunningTime="2026-11-03 21:42:55.138180385 +0000 UTC m=+3276.769493861"
Nov 03 21:42:56.036028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:56.035990    5533 generic.go:356] "Generic (PLEG): container finished" podID="2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" containerID="71a8ca671d926d000740526fbd7bf4ec1ff195a7ed82c07ab447c363054edfe9" exitCode=0
Nov 03 21:42:56.036391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:56.036072    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh" event={"ID":"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf","Type":"ContainerDied","Data":"71a8ca671d926d000740526fbd7bf4ec1ff195a7ed82c07ab447c363054edfe9"}
Nov 03 21:42:57.186357 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.186333    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:57.203771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.203747    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\") pod \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") "
Nov 03 21:42:57.203891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.203878    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\" (UniqueName: \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\") pod \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") "
Nov 03 21:42:57.203933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.203913    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\") pod \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\" (UID: \"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf\") "
Nov 03 21:42:57.204312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.204289    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca" pod "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" (UID: "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:57.205790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.205768    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca" pod "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" (UID: "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:42:57.205877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.205837    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn" pod "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" (UID: "2ce8cb23-5f00-4d35-b5c8-777c91f71fdf"). InnerVolumeSpecName "kube-api-access-9bvzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:57.305603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.305539    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bvzn\" (UniqueName: \"kubernetes.io/projected/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-kube-api-access-9bvzn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:57.305603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.305571    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:57.305603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:57.305588    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:42:58.051195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:58.051167    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"
Nov 03 21:42:58.051388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:58.051211    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh" event={"ID":"2ce8cb23-5f00-4d35-b5c8-777c91f71fdf","Type":"ContainerDied","Data":"6026da1d608a90fac868808008aa3affabe63abf1da2542797b46c9d1c339865"}
Nov 03 21:42:58.051388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:42:58.051269    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6026da1d608a90fac868808008aa3affabe63abf1da2542797b46c9d1c339865"
Nov 03 21:43:00.992350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:00.992319    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:43:00.992829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:00.992384    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:43:04.022300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:04.022268    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:43:04.022736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:04.022348    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:43:09.067374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:09.067337    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:43:19.106163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:19.106123    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:43:19.119934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:19.119912    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:43:19.143586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:19.143561    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:43:19.154924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:19.154903    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:43:19.261918 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:43:19.261887    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:43:25.510741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:25.510707    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:43:38.670351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.670313    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:43:38.671193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.671170    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" containerName="revoke-keys"
Nov 03 21:43:38.671193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.671192    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" containerName="revoke-keys"
Nov 03 21:43:38.671359 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.671345    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" containerName="revoke-keys"
Nov 03 21:43:38.676370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.676349    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.725851 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.725829    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a02404-openshift-default-dockercfg-slz6p\""
Nov 03 21:43:38.753374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753346    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753390    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753442    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753554    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753628    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753701    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753834 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753774    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9xb\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.753942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.753915    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.754022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.754001    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.776415 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.776393    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:43:38.855617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855583    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855674    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855719    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855754    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9xb\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855804    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855847    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.855936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855928    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.855973    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856008    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856040    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856355    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856502    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856508    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.856857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.856832    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.858323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.858299    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.858930 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.858907    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.916924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.916894    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.919755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.919725    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9xb\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\") pod \"e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") " pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:38.989868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:38.989776    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:39.422877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:39.422849    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:43:39.423707 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:43:39.423675    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cf8c53_6af5_41aa_aa39_8aced313cb23.slice/crio-13ce8bb8900f20fe588bb0584fedebaf4044a3899685cda7159f48456243d87a WatchSource:0}: Error finding container 13ce8bb8900f20fe588bb0584fedebaf4044a3899685cda7159f48456243d87a: Status 404 returned error can't find the container with id 13ce8bb8900f20fe588bb0584fedebaf4044a3899685cda7159f48456243d87a
Nov 03 21:43:39.426207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:39.426172    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:39.426294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:39.426242    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:39.426294 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:39.426279    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:40.368811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.368775    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerStarted","Data":"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"}
Nov 03 21:43:40.369218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.368835    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerStarted","Data":"13ce8bb8900f20fe588bb0584fedebaf4044a3899685cda7159f48456243d87a"}
Nov 03 21:43:40.530833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.530789    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" podUID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" containerName="main" containerID="cri-o://d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a" gracePeriod=45
Nov 03 21:43:40.538799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.538753    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podStartSLOduration=2.538740152 podStartE2EDuration="2.538740152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:40.536683424 +0000 UTC m=+3322.167996902" watchObservedRunningTime="2026-11-03 21:43:40.538740152 +0000 UTC m=+3322.170053639"
Nov 03 21:43:40.894232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.894196    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:43:40.982405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982371    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.982549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982434    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.982549 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982485    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.982718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982613    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\" (UniqueName: \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.982718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982683    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.982841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982760    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:40.982841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982778    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:40.982917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982889    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:40.982960 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.982942    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\") pod \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\" (UID: \"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b\") "
Nov 03 21:43:40.983434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.983366    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:40.983434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.983390    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:40.983434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.983404    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:40.985197 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.985168    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:43:40.985295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.985233    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:43:40.985347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.985314    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m" pod "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" (UID: "c0bb402b-6b0c-4fcd-9637-9b35f1153e3b"). InnerVolumeSpecName "kube-api-access-z4z2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:40.990498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.990474    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:40.996364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:40.996344    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:41.085124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.085090    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:41.085124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.085124    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:41.085310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.085139    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4z2m\" (UniqueName: \"kubernetes.io/projected/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b-kube-api-access-z4z2m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:41.379146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379059    5533 generic.go:356] "Generic (PLEG): container finished" podID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" containerID="d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a" exitCode=0
Nov 03 21:43:41.379146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379115    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"
Nov 03 21:43:41.379744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379145    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" event={"ID":"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b","Type":"ContainerDied","Data":"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"}
Nov 03 21:43:41.379744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379190    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp" event={"ID":"c0bb402b-6b0c-4fcd-9637-9b35f1153e3b","Type":"ContainerDied","Data":"deb219d10a8856bb043831d3a7830ae4484b9853114acd4275e9be4aa65f1fe7"}
Nov 03 21:43:41.379744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379211    5533 scope.go:122] "RemoveContainer" containerID="d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"
Nov 03 21:43:41.379909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.379773    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:41.381148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.381125    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:43:41.390931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.390914    5533 scope.go:122] "RemoveContainer" containerID="d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"
Nov 03 21:43:41.391335 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:43:41.391208    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a\": container with ID starting with d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a not found: ID does not exist" containerID="d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"
Nov 03 21:43:41.391335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.391240    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a"} err="failed to get container status \"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a\": rpc error: code = NotFound desc = could not find container \"d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a\": container with ID starting with d6d72ab1dd265ecced92d1acf370bdcadd157bca079470dee6b9572103aff58a not found: ID does not exist"
Nov 03 21:43:41.660343 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.660251    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:43:41.666169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.666142    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-4d1799/auth-test-model-4d1799-kserve-7bd9f759cc-mnlwp"]
Nov 03 21:43:41.742374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.742340    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:43:41.743023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.743008    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" containerName="main"
Nov 03 21:43:41.743023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.743024    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" containerName="main"
Nov 03 21:43:41.743145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.743114    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" containerName="main"
Nov 03 21:43:41.748004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.747986    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.799662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.799514    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7da0f7-openshift-default-dockercfg-h89nd\""
Nov 03 21:43:41.821657 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.819014    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:43:41.906346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.906297    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrdd\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.907810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.907680    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.907810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.907765    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.907814    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.907926    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908022 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.907973    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.908021    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.908070    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:41.908218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:41.908125    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.009962    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010038    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010099    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010216    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrdd\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010291    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010336    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010370    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010431    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.010673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.010459    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.013328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.011863    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.013328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.012178    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.013328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.012365    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.013328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.012878    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.013328 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.013254    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.016112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.016066    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.016574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.016549    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.069719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.069691    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.069883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.069863    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrdd\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.094183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.094161    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:42.493660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.493607    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bb402b-6b0c-4fcd-9637-9b35f1153e3b" path="/var/lib/kubelet/pods/c0bb402b-6b0c-4fcd-9637-9b35f1153e3b/volumes"
Nov 03 21:43:42.518856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.518830    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:43:42.519545 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:43:42.519521    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1999725a_e166_4784_b3c0_60d11a2bc0fc.slice/crio-793efe79f7842087e480caa1eec8def8d13e1f74a0366fe9edd7bd4beff3307d WatchSource:0}: Error finding container 793efe79f7842087e480caa1eec8def8d13e1f74a0366fe9edd7bd4beff3307d: Status 404 returned error can't find the container with id 793efe79f7842087e480caa1eec8def8d13e1f74a0366fe9edd7bd4beff3307d
Nov 03 21:43:42.522866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.522820    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:42.522997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.522971    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:42.523088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:42.523066    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:43:43.403034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:43.402995    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerStarted","Data":"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"}
Nov 03 21:43:43.403251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:43.403088    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerStarted","Data":"793efe79f7842087e480caa1eec8def8d13e1f74a0366fe9edd7bd4beff3307d"}
Nov 03 21:43:43.533493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:43.533442    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podStartSLOduration=2.533427498 podStartE2EDuration="2.533427498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:43.531035437 +0000 UTC m=+3325.162348914" watchObservedRunningTime="2026-11-03 21:43:43.533427498 +0000 UTC m=+3325.164740974"
Nov 03 21:43:44.095078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:44.095037    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:44.100890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:44.100862    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:44.411093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:44.411014    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:44.412053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:44.412028    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:43:49.264917 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:43:49.264881    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:43:50.517161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:50.517125    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:43:52.130009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.129976    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:43:52.133824 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.133804    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.235667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.235039    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a02404-dockercfg-vncm9\""
Nov 03 21:43:52.235667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.235134    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a02404\""
Nov 03 21:43:52.243655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.242718    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:43:52.248672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.247159    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrc9\" (UniqueName: \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.248672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.247410    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.348358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.348318    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrc9\" (UniqueName: \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.348560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.348541    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.349504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.349478    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.406558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.406484    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrc9\" (UniqueName: \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\") pod \"payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.448101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.448071    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:52.849817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:52.849784    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:43:52.850090 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:43:52.850064    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71f5f3d_1f05_4331_93d7_96c27db86cae.slice/crio-a453a14b1dcf3202cbe752eb9c236f63a6d55b6555dbee9b271e58164607a1eb WatchSource:0}: Error finding container a453a14b1dcf3202cbe752eb9c236f63a6d55b6555dbee9b271e58164607a1eb: Status 404 returned error can't find the container with id a453a14b1dcf3202cbe752eb9c236f63a6d55b6555dbee9b271e58164607a1eb
Nov 03 21:43:53.496709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:53.496675    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" event={"ID":"e71f5f3d-1f05-4331-93d7-96c27db86cae","Type":"ContainerStarted","Data":"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"}
Nov 03 21:43:53.497125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:53.496723    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" event={"ID":"e71f5f3d-1f05-4331-93d7-96c27db86cae","Type":"ContainerStarted","Data":"a453a14b1dcf3202cbe752eb9c236f63a6d55b6555dbee9b271e58164607a1eb"}
Nov 03 21:43:53.497125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:53.496827    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:43:53.607806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:53.607759    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" podStartSLOduration=1.607746581 podStartE2EDuration="1.607746581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:53.605947417 +0000 UTC m=+3335.237260894" watchObservedRunningTime="2026-11-03 21:43:53.607746581 +0000 UTC m=+3335.239060057"
Nov 03 21:43:54.747649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.747600    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:43:54.766891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.766866    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:54.829681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.820564    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7da0f7-dockercfg-kzxn8\""
Nov 03 21:43:54.829681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.820710    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7da0f7\""
Nov 03 21:43:54.837656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.834306    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:43:54.906660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.906269    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7c7\" (UniqueName: \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:54.906660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.906441    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:54.981692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.981661    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:43:54.986238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:54.986218    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.007243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.007194    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.007339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.007264    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:55.007394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.007373    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7c7\" (UniqueName: \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:55.007440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.007405    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmpg\" (UniqueName: \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.007995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.007977    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:55.029420 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.029397    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:43:55.121383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.118574    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmpg\" (UniqueName: \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.121383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.118740    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.121383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.120891    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.124093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.124070    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7c7\" (UniqueName: \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\") pod \"payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") " pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:55.181680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.181620    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmpg\" (UniqueName: \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\") pod \"payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.300790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.300725    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:55.403083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.391758    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:55.698114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.698074    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:43:55.699420 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:43:55.699395    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27685cb_4b7a_4e40_ad60_2fa4069eb673.slice/crio-932f6f433e99ca5ac8e7cb53aa7745e131f9eab967061f6e71f20bb98c469c3f WatchSource:0}: Error finding container 932f6f433e99ca5ac8e7cb53aa7745e131f9eab967061f6e71f20bb98c469c3f: Status 404 returned error can't find the container with id 932f6f433e99ca5ac8e7cb53aa7745e131f9eab967061f6e71f20bb98c469c3f
Nov 03 21:43:55.831250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:55.831223    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:43:55.832066 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:43:55.832006    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d487f7e_c3af_4363_869f_1fa3a43220d4.slice/crio-75af3f190716476d9e5fe3249694e382e421279fcf4f3c0de467bdb17caa543d WatchSource:0}: Error finding container 75af3f190716476d9e5fe3249694e382e421279fcf4f3c0de467bdb17caa543d: Status 404 returned error can't find the container with id 75af3f190716476d9e5fe3249694e382e421279fcf4f3c0de467bdb17caa543d
Nov 03 21:43:56.528801 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.528767    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" event={"ID":"6d487f7e-c3af-4363-869f-1fa3a43220d4","Type":"ContainerStarted","Data":"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"}
Nov 03 21:43:56.528973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.528836    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:43:56.528973 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.528863    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" event={"ID":"6d487f7e-c3af-4363-869f-1fa3a43220d4","Type":"ContainerStarted","Data":"75af3f190716476d9e5fe3249694e382e421279fcf4f3c0de467bdb17caa543d"}
Nov 03 21:43:56.530388 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.530356    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" event={"ID":"c27685cb-4b7a-4e40-ad60-2fa4069eb673","Type":"ContainerStarted","Data":"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"}
Nov 03 21:43:56.530499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.530411    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" event={"ID":"c27685cb-4b7a-4e40-ad60-2fa4069eb673","Type":"ContainerStarted","Data":"932f6f433e99ca5ac8e7cb53aa7745e131f9eab967061f6e71f20bb98c469c3f"}
Nov 03 21:43:56.530499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.530441    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:43:56.646684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.646612    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" podStartSLOduration=2.646596847 podStartE2EDuration="2.646596847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:56.643723922 +0000 UTC m=+3338.275037400" watchObservedRunningTime="2026-11-03 21:43:56.646596847 +0000 UTC m=+3338.277910329"
Nov 03 21:43:56.769656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:56.762915    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" podStartSLOduration=2.762893524 podStartE2EDuration="2.762893524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:56.760368238 +0000 UTC m=+3338.391681718" watchObservedRunningTime="2026-11-03 21:43:56.762893524 +0000 UTC m=+3338.394207002"
Nov 03 21:43:58.046732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:58.046695    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"]
Nov 03 21:43:58.095173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:58.095135    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb30a63-qn8tc"]
Nov 03 21:43:58.497797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:43:58.497758    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cddb9d6-5740-481a-9f07-503aca0ea8cf" path="/var/lib/kubelet/pods/2cddb9d6-5740-481a-9f07-503aca0ea8cf/volumes"
Nov 03 21:44:00.722215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:00.720781    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": read tcp 10.133.0.2:41668->10.133.0.251:15021: read: connection reset by peer" start-of-body=
Nov 03 21:44:00.722215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:00.720844    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": read tcp 10.133.0.2:41668->10.133.0.251:15021: read: connection reset by peer"
Nov 03 21:44:01.597895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.597866    5533 generic.go:356] "Generic (PLEG): container finished" podID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerID="67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73" exitCode=0
Nov 03 21:44:01.598092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.597939    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerDied","Data":"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"}
Nov 03 21:44:01.598321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.598306    5533 scope.go:122] "RemoveContainer" containerID="67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"
Nov 03 21:44:01.600067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.600040    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:44:01.600157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.600111    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:44:01.600157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:01.600142    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:44:02.608369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:02.608329    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerStarted","Data":"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"}
Nov 03 21:44:03.497315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:03.497279    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:44:03.497497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:03.497477    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 21:44:03.497544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:03.497516    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 21:44:04.497994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:04.497954    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 21:44:04.498407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:04.498019    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 21:44:04.515522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:04.515499    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:44:05.497431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:05.497395    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 21:44:05.497599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:05.497451    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 21:44:06.497750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:06.497718    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 21:44:06.498216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:06.497773    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 21:44:07.497438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:07.497405    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 21:44:07.497614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:07.497466    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 21:44:07.542170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:07.542143    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:44:07.542547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:07.542206    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:44:08.502083 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:08.502056    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:44:08.502538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:08.502521    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:44:08.503261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:08.503237    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:44:09.046673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:09.046631    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"]
Nov 03 21:44:09.050262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:09.050235    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e518e00c-sjjj6"]
Nov 03 21:44:10.494890 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:10.494857    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1090cf3-7aeb-4ae7-824a-634127ab27b3" path="/var/lib/kubelet/pods/b1090cf3-7aeb-4ae7-824a-634127ab27b3/volumes"
Nov 03 21:44:19.262204 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:44:19.262166    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:44:24.929260 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:24.929074    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:44:43.047392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:43.047356    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"]
Nov 03 21:44:43.093975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:43.093945    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edb5ae7-5rv5l"]
Nov 03 21:44:44.495431 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:44.495401    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24af39d9-bbcd-4813-9dd6-7426152e959d" path="/var/lib/kubelet/pods/24af39d9-bbcd-4813-9dd6-7426152e959d/volumes"
Nov 03 21:44:49.262750 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:44:49.262714    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:44:54.046891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:54.046859    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"]
Nov 03 21:44:54.093149 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:54.093117    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74094f44-rsg2j"]
Nov 03 21:44:54.497411 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:44:54.497380    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548f433f-0632-479f-8bed-c4409f3f2194" path="/var/lib/kubelet/pods/548f433f-0632-479f-8bed-c4409f3f2194/volumes"
Nov 03 21:45:00.162251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.162221    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:00.166723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.166702    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:00.166878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.166812    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.170728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.170706    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:00.170859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.170823    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.174699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.174679    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.207625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.207603    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:00.211517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.211499    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:00.211654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.211620    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.215292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.215273    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:00.215384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.215367    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.219318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.219299    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:00.219718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.219702    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.234536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.234514    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:00.234748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.234724    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.239400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.239380    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:00.239491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.239478    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.243339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.243319    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:00.243693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.243603    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.249039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.249023    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.251844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.251823    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:45:00.254891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.254875    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:00.264362 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.264342    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:00.264530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.264510    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.269412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.269394    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:00.269511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.269433    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:00.269511 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.269492    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.304723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.304303    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:00.313345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.313318    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:00.314772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.314749    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:00.317872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.317850    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:00.320830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.320798    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:00.344097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344062    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxbm\" (UniqueName: \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.344216 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344118    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.344290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344217    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6pc2\" (UniqueName: \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.344290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344258    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.344471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344297    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.344471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344331    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mw69\" (UniqueName: \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.344471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344377    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.344471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344420    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.344471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344468    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344500    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344566    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344596    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5qt\" (UniqueName: \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344654    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkhx\" (UniqueName: \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344683    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxw5m\" (UniqueName: \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.344719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344716    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577nd\" (UniqueName: \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.344967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344750    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.344967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344839    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4glpl\" (UniqueName: \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.344967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.344905    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhpp\" (UniqueName: \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.360219 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.359700    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:00.361745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.361719    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:00.364906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.364884    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:00.366786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.366769    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:00.418676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.418608    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:00.445697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445670    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.445800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445711    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.445800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445739    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.445876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445797    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.445876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445829    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.445876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445866    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.446003 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445897    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslg9\" (UniqueName: \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.446003 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.445932    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.446107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446011    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkhx\" (UniqueName: \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.446107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446067    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577nd\" (UniqueName: \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.446218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446104    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.446218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446134    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.446218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446186    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4glpl\" (UniqueName: \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.446435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446226    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxbm\" (UniqueName: \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.446435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446259    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6pc2\" (UniqueName: \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.446435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446303    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mw69\" (UniqueName: \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.446435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446360    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.446435 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446424    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvkc\" (UniqueName: \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446455    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446484    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5qt\" (UniqueName: \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446530    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxw5m\" (UniqueName: \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446551    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446576    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446628    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhdl\" (UniqueName: \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446664    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446674    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446708    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446707    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.446752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.446719    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhpp\" (UniqueName: \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.447412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.447389    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.447475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.447389    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.447475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.447465    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.447593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.447572    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.505536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.505434    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6pc2\" (UniqueName: \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.505536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.505477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577nd\" (UniqueName: \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.506052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.506021    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhpp\" (UniqueName: \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.506482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.506452    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mw69\" (UniqueName: \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.506594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.506486    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4glpl\" (UniqueName: \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.506695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.506596    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkhx\" (UniqueName: \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.506832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.506814    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxw5m\" (UniqueName: \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.507145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.507127    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxbm\" (UniqueName: \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\") pod \"maas-api-key-cleanup-29895705-nwh7l\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.507246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.507230    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5qt\" (UniqueName: \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.538712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.538690    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:00.545978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.545958    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:00.547598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.547571    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.547719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.547628    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhdl\" (UniqueName: \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.547719 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.547703    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.547845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.547784    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslg9\" (UniqueName: \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.547955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.547862    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.548024 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.548005    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvkc\" (UniqueName: \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.548422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.548393    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.548557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.548531    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.548763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.548736    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.553227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.553208    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:00.565301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.565271    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:00.572710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.572479    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:00.580937 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.580764    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:00.620833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.620218    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhdl\" (UniqueName: \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.640480 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.640419    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvkc\" (UniqueName: \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.680151 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.680119    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslg9\" (UniqueName: \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.786875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.786842    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:00.799426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.799388    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:00.802768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.802570    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:00.889474 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.889248    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:00.909672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.903222    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:00.918908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.918838    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:00.946933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.946906    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:00.985017 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.984984    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:00.991551 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:00.991470    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:00.996184 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:00.994342    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4a74c2_1cbe_4f55_b104_146db6e3095d.slice/crio-ead6d6f8af24b12f601a7b67d204a4aef4a02b2d34ad1c00eda2e3f5388afd85 WatchSource:0}: Error finding container ead6d6f8af24b12f601a7b67d204a4aef4a02b2d34ad1c00eda2e3f5388afd85: Status 404 returned error can't find the container with id ead6d6f8af24b12f601a7b67d204a4aef4a02b2d34ad1c00eda2e3f5388afd85
Nov 03 21:45:01.002097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.002024    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:01.060773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.059507    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:01.064383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.064309    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:01.091686 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.089323    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84805bc0_e9ad_49a0_9cfe_da316107c256.slice/crio-38b4ca92fb7a2837bec75a275594ac059cd4adc1d5722d8b4992962af1e833b1 WatchSource:0}: Error finding container 38b4ca92fb7a2837bec75a275594ac059cd4adc1d5722d8b4992962af1e833b1: Status 404 returned error can't find the container with id 38b4ca92fb7a2837bec75a275594ac059cd4adc1d5722d8b4992962af1e833b1
Nov 03 21:45:01.108628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.108564    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerStarted","Data":"95f92da24bbf450fe998b2092fbb2497ea8944b2bc719ec60cc320d067495e6d"}
Nov 03 21:45:01.110164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.110099    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerStarted","Data":"ead6d6f8af24b12f601a7b67d204a4aef4a02b2d34ad1c00eda2e3f5388afd85"}
Nov 03 21:45:01.117020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.116675    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerStarted","Data":"fb8099e8275df4c73d4a2f0ddf8441fe0db3a806d22d3f8cf05286db254c660e"}
Nov 03 21:45:01.119358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.119335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerStarted","Data":"5fc44c7c4db9c7c734c527bb704ccad780ce677ea01d60f2abf17ce6566ce1c0"}
Nov 03 21:45:01.205846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.205782    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:01.209016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.208998    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:01.240071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.240050    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:01.347278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.347250    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:01.352967 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.352939    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a2b3ff_9c0b_4ccd_869d_26a1a943b55d.slice/crio-062b7d5236020d05a04a2994a257bca248270286db0c68edad47841210e6e0f6 WatchSource:0}: Error finding container 062b7d5236020d05a04a2994a257bca248270286db0c68edad47841210e6e0f6: Status 404 returned error can't find the container with id 062b7d5236020d05a04a2994a257bca248270286db0c68edad47841210e6e0f6
Nov 03 21:45:01.355067 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.355013    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0419eda7_4ff1_4caf_a989_c56e34236958.slice/crio-85a2000224eb6f018809337f63f2b86b6687a1cfc0ced8743f89ee687fe45385 WatchSource:0}: Error finding container 85a2000224eb6f018809337f63f2b86b6687a1cfc0ced8743f89ee687fe45385: Status 404 returned error can't find the container with id 85a2000224eb6f018809337f63f2b86b6687a1cfc0ced8743f89ee687fe45385
Nov 03 21:45:01.356299 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.356275    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158ab112_364a_4cf3_80ea_d21808276177.slice/crio-893a48b3245de68b8b2a93e441713ba66d72bcce63d42023f92f8207a8c2efaf WatchSource:0}: Error finding container 893a48b3245de68b8b2a93e441713ba66d72bcce63d42023f92f8207a8c2efaf: Status 404 returned error can't find the container with id 893a48b3245de68b8b2a93e441713ba66d72bcce63d42023f92f8207a8c2efaf
Nov 03 21:45:01.358278 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.358176    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0999f1d3_6a28_4a9a_822f_981ef6dea956.slice/crio-bf5c3c3456c25a6f17b345a576dab1678734859b5ea6938639b62f14a00124ab WatchSource:0}: Error finding container bf5c3c3456c25a6f17b345a576dab1678734859b5ea6938639b62f14a00124ab: Status 404 returned error can't find the container with id bf5c3c3456c25a6f17b345a576dab1678734859b5ea6938639b62f14a00124ab
Nov 03 21:45:01.382068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.382043    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:01.384520 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.384491    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e90c9c_b003_44b3_8f7e_97be647173f6.slice/crio-aa0c7d631810a53d7b642fca06f0a6bc296ccacd95f90a1392c14171b5f745de WatchSource:0}: Error finding container aa0c7d631810a53d7b642fca06f0a6bc296ccacd95f90a1392c14171b5f745de: Status 404 returned error can't find the container with id aa0c7d631810a53d7b642fca06f0a6bc296ccacd95f90a1392c14171b5f745de
Nov 03 21:45:01.411496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:01.411474    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:01.459562 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:45:01.459524    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059e5404_273a_443a_9656_387e5130f21b.slice/crio-3cc7b571ddcaefb3130e8d4abfc43727629eb9e06ad570ca22ecdd8895ac66aa WatchSource:0}: Error finding container 3cc7b571ddcaefb3130e8d4abfc43727629eb9e06ad570ca22ecdd8895ac66aa: Status 404 returned error can't find the container with id 3cc7b571ddcaefb3130e8d4abfc43727629eb9e06ad570ca22ecdd8895ac66aa
Nov 03 21:45:02.135684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.135630    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerStarted","Data":"f08ee17d17397fb3c2f9c6e14d8b7476ab5937385e7b47b1ac671ed27d1f3165"}
Nov 03 21:45:02.137071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.137047    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerStarted","Data":"3cc7b571ddcaefb3130e8d4abfc43727629eb9e06ad570ca22ecdd8895ac66aa"}
Nov 03 21:45:02.138733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.138704    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerStarted","Data":"a6802f667952bdc1d50cf7f831794c95893b5e3814804c632d5c74b87c2b75b2"}
Nov 03 21:45:02.140230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.140204    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerStarted","Data":"cd365f2ffaed236f5bb48d184811ccf0f7ea916d8f1e81f6290b89efb4fb8db9"}
Nov 03 21:45:02.140342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.140253    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerStarted","Data":"85a2000224eb6f018809337f63f2b86b6687a1cfc0ced8743f89ee687fe45385"}
Nov 03 21:45:02.141540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.141521    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerStarted","Data":"062b7d5236020d05a04a2994a257bca248270286db0c68edad47841210e6e0f6"}
Nov 03 21:45:02.143107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.143073    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerStarted","Data":"830823b7cd840cd59cae66a1c31a7bcf14b65d443699b9197da8286d2e063603"}
Nov 03 21:45:02.143210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.143125    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerStarted","Data":"893a48b3245de68b8b2a93e441713ba66d72bcce63d42023f92f8207a8c2efaf"}
Nov 03 21:45:02.144864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.144836    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerStarted","Data":"38b4ca92fb7a2837bec75a275594ac059cd4adc1d5722d8b4992962af1e833b1"}
Nov 03 21:45:02.146403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.146378    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerStarted","Data":"f23fe142ad485fdbd7bf867390f69c9a0f5fcbe191085cfee47639075756e892"}
Nov 03 21:45:02.146524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.146422    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerStarted","Data":"bf5c3c3456c25a6f17b345a576dab1678734859b5ea6938639b62f14a00124ab"}
Nov 03 21:45:02.147590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.147568    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerStarted","Data":"aa0c7d631810a53d7b642fca06f0a6bc296ccacd95f90a1392c14171b5f745de"}
Nov 03 21:45:02.148955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.148934    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerStarted","Data":"1e61ffe9ee8b3547402ad31cf13ec857f38868a8bd36aaa61c32681640803827"}
Nov 03 21:45:02.150433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.150407    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerStarted","Data":"6a902a9e09defdf717ce8952fb21aeb817a339763f35d9a2d3462741feb22a62"}
Nov 03 21:45:02.150524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.150450    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerStarted","Data":"986fbc3f82b685954331d3a1a3da2620041f4a0cd6a405a9b815836cf58611a2"}
Nov 03 21:45:02.152394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.152367    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerStarted","Data":"5531a9a75efc3844d27cc5c6b8fdf0614aa11828599e62c32ac5a203d6be82f2"}
Nov 03 21:45:02.248032 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.247984    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" podStartSLOduration=2.24797094 podStartE2EDuration="2.24797094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.245193085 +0000 UTC m=+3403.876506563" watchObservedRunningTime="2026-11-03 21:45:02.24797094 +0000 UTC m=+3403.879284414"
Nov 03 21:45:02.359237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.355720    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" podStartSLOduration=2.355703241 podStartE2EDuration="2.355703241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.355417648 +0000 UTC m=+3403.986731125" watchObservedRunningTime="2026-11-03 21:45:02.355703241 +0000 UTC m=+3403.987016720"
Nov 03 21:45:02.464764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.464715    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" podStartSLOduration=2.464702056 podStartE2EDuration="2.464702056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.462757224 +0000 UTC m=+3404.094070700" watchObservedRunningTime="2026-11-03 21:45:02.464702056 +0000 UTC m=+3404.096015524"
Nov 03 21:45:02.572247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.572203    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" podStartSLOduration=2.572188889 podStartE2EDuration="2.572188889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.570195129 +0000 UTC m=+3404.201508610" watchObservedRunningTime="2026-11-03 21:45:02.572188889 +0000 UTC m=+3404.203502365"
Nov 03 21:45:02.679600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.679556    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" podStartSLOduration=2.679541145 podStartE2EDuration="2.679541145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.678120137 +0000 UTC m=+3404.309433614" watchObservedRunningTime="2026-11-03 21:45:02.679541145 +0000 UTC m=+3404.310854621"
Nov 03 21:45:02.790749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.790705    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" podStartSLOduration=2.7906918469999997 podStartE2EDuration="2.790691847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.787679483 +0000 UTC m=+3404.418992978" watchObservedRunningTime="2026-11-03 21:45:02.790691847 +0000 UTC m=+3404.422005324"
Nov 03 21:45:02.900955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:02.900905    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" podStartSLOduration=2.900892727 podStartE2EDuration="2.900892727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.897583419 +0000 UTC m=+3404.528896935" watchObservedRunningTime="2026-11-03 21:45:02.900892727 +0000 UTC m=+3404.532206203"
Nov 03 21:45:03.009883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.009840    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" podStartSLOduration=3.009826344 podStartE2EDuration="3.009826344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:03.007462914 +0000 UTC m=+3404.638776391" watchObservedRunningTime="2026-11-03 21:45:03.009826344 +0000 UTC m=+3404.641139821"
Nov 03 21:45:03.162531 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.162503    5533 generic.go:356] "Generic (PLEG): container finished" podID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerID="1e61ffe9ee8b3547402ad31cf13ec857f38868a8bd36aaa61c32681640803827" exitCode=60
Nov 03 21:45:03.162654 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.162583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerDied","Data":"1e61ffe9ee8b3547402ad31cf13ec857f38868a8bd36aaa61c32681640803827"}
Nov 03 21:45:03.162839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.162821    5533 scope.go:122] "RemoveContainer" containerID="1e61ffe9ee8b3547402ad31cf13ec857f38868a8bd36aaa61c32681640803827"
Nov 03 21:45:03.164448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.164380    5533 generic.go:356] "Generic (PLEG): container finished" podID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerID="6a902a9e09defdf717ce8952fb21aeb817a339763f35d9a2d3462741feb22a62" exitCode=60
Nov 03 21:45:03.164520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.164453    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerDied","Data":"6a902a9e09defdf717ce8952fb21aeb817a339763f35d9a2d3462741feb22a62"}
Nov 03 21:45:03.164758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.164740    5533 scope.go:122] "RemoveContainer" containerID="6a902a9e09defdf717ce8952fb21aeb817a339763f35d9a2d3462741feb22a62"
Nov 03 21:45:03.166449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.166430    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerID="5531a9a75efc3844d27cc5c6b8fdf0614aa11828599e62c32ac5a203d6be82f2" exitCode=60
Nov 03 21:45:03.166626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.166570    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerDied","Data":"5531a9a75efc3844d27cc5c6b8fdf0614aa11828599e62c32ac5a203d6be82f2"}
Nov 03 21:45:03.166809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.166794    5533 scope.go:122] "RemoveContainer" containerID="5531a9a75efc3844d27cc5c6b8fdf0614aa11828599e62c32ac5a203d6be82f2"
Nov 03 21:45:03.168185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.168165    5533 generic.go:356] "Generic (PLEG): container finished" podID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerID="f08ee17d17397fb3c2f9c6e14d8b7476ab5937385e7b47b1ac671ed27d1f3165" exitCode=60
Nov 03 21:45:03.168248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.168234    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerDied","Data":"f08ee17d17397fb3c2f9c6e14d8b7476ab5937385e7b47b1ac671ed27d1f3165"}
Nov 03 21:45:03.168430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.168413    5533 scope.go:122] "RemoveContainer" containerID="f08ee17d17397fb3c2f9c6e14d8b7476ab5937385e7b47b1ac671ed27d1f3165"
Nov 03 21:45:03.177491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.177470    5533 generic.go:356] "Generic (PLEG): container finished" podID="059e5404-273a-443a-9656-387e5130f21b" containerID="90d8f459a1828e74caf0fe737e08ca9ce467bc0b60cc3dde1b19d724e5c54532" exitCode=60
Nov 03 21:45:03.177675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.177627    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerDied","Data":"90d8f459a1828e74caf0fe737e08ca9ce467bc0b60cc3dde1b19d724e5c54532"}
Nov 03 21:45:03.177873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.177841    5533 scope.go:122] "RemoveContainer" containerID="90d8f459a1828e74caf0fe737e08ca9ce467bc0b60cc3dde1b19d724e5c54532"
Nov 03 21:45:03.179608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.179588    5533 generic.go:356] "Generic (PLEG): container finished" podID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerID="a6802f667952bdc1d50cf7f831794c95893b5e3814804c632d5c74b87c2b75b2" exitCode=60
Nov 03 21:45:03.179693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.179661    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerDied","Data":"a6802f667952bdc1d50cf7f831794c95893b5e3814804c632d5c74b87c2b75b2"}
Nov 03 21:45:03.179856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.179841    5533 scope.go:122] "RemoveContainer" containerID="a6802f667952bdc1d50cf7f831794c95893b5e3814804c632d5c74b87c2b75b2"
Nov 03 21:45:03.181233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.181212    5533 generic.go:356] "Generic (PLEG): container finished" podID="0419eda7-4ff1-4caf-a989-c56e34236958" containerID="cd365f2ffaed236f5bb48d184811ccf0f7ea916d8f1e81f6290b89efb4fb8db9" exitCode=60
Nov 03 21:45:03.181366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.181312    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerDied","Data":"cd365f2ffaed236f5bb48d184811ccf0f7ea916d8f1e81f6290b89efb4fb8db9"}
Nov 03 21:45:03.181496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.181479    5533 scope.go:122] "RemoveContainer" containerID="cd365f2ffaed236f5bb48d184811ccf0f7ea916d8f1e81f6290b89efb4fb8db9"
Nov 03 21:45:03.184108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.184087    5533 generic.go:356] "Generic (PLEG): container finished" podID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerID="433dc0362378d6d8b379ac96611f41b2ebc0a451129ac59492ef463ca84ee726" exitCode=60
Nov 03 21:45:03.184176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.184161    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerDied","Data":"433dc0362378d6d8b379ac96611f41b2ebc0a451129ac59492ef463ca84ee726"}
Nov 03 21:45:03.185438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.185419    5533 scope.go:122] "RemoveContainer" containerID="433dc0362378d6d8b379ac96611f41b2ebc0a451129ac59492ef463ca84ee726"
Nov 03 21:45:03.187466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.187448    5533 generic.go:356] "Generic (PLEG): container finished" podID="158ab112-364a-4cf3-80ea-d21808276177" containerID="830823b7cd840cd59cae66a1c31a7bcf14b65d443699b9197da8286d2e063603" exitCode=60
Nov 03 21:45:03.187539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.187524    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerDied","Data":"830823b7cd840cd59cae66a1c31a7bcf14b65d443699b9197da8286d2e063603"}
Nov 03 21:45:03.187822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.187803    5533 scope.go:122] "RemoveContainer" containerID="830823b7cd840cd59cae66a1c31a7bcf14b65d443699b9197da8286d2e063603"
Nov 03 21:45:03.192005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.191980    5533 generic.go:356] "Generic (PLEG): container finished" podID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerID="fd9b58def8de803864adf5afd700ad47d4aa658a94408b1a29cf7c3b45a991de" exitCode=60
Nov 03 21:45:03.192116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.192097    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerDied","Data":"fd9b58def8de803864adf5afd700ad47d4aa658a94408b1a29cf7c3b45a991de"}
Nov 03 21:45:03.192555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.192532    5533 scope.go:122] "RemoveContainer" containerID="fd9b58def8de803864adf5afd700ad47d4aa658a94408b1a29cf7c3b45a991de"
Nov 03 21:45:03.205884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.205807    5533 generic.go:356] "Generic (PLEG): container finished" podID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerID="f23fe142ad485fdbd7bf867390f69c9a0f5fcbe191085cfee47639075756e892" exitCode=60
Nov 03 21:45:03.205987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.205900    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerDied","Data":"f23fe142ad485fdbd7bf867390f69c9a0f5fcbe191085cfee47639075756e892"}
Nov 03 21:45:03.206211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.206189    5533 scope.go:122] "RemoveContainer" containerID="f23fe142ad485fdbd7bf867390f69c9a0f5fcbe191085cfee47639075756e892"
Nov 03 21:45:03.219545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.219514    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerID="5f5d3e26f5d2ba1a173e817f2f1619239711c8e4b46a49e421fc4dfa643713b7" exitCode=60
Nov 03 21:45:03.219674 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.219587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerDied","Data":"5f5d3e26f5d2ba1a173e817f2f1619239711c8e4b46a49e421fc4dfa643713b7"}
Nov 03 21:45:03.219866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:03.219846    5533 scope.go:122] "RemoveContainer" containerID="5f5d3e26f5d2ba1a173e817f2f1619239711c8e4b46a49e421fc4dfa643713b7"
Nov 03 21:45:04.229766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.229648    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerStarted","Data":"cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"}
Nov 03 21:45:04.231979 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.231952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerStarted","Data":"b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"}
Nov 03 21:45:04.233902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.233868    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerStarted","Data":"3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"}
Nov 03 21:45:04.235716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.235691    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerStarted","Data":"0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"}
Nov 03 21:45:04.237555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.237529    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerStarted","Data":"2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"}
Nov 03 21:45:04.239465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.239444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerStarted","Data":"52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"}
Nov 03 21:45:04.241356 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.241336    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerStarted","Data":"743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"}
Nov 03 21:45:04.243401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.243377    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerStarted","Data":"4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"}
Nov 03 21:45:04.245148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.245124    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerStarted","Data":"2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"}
Nov 03 21:45:04.246898 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.246879    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerStarted","Data":"1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"}
Nov 03 21:45:04.248647 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.248615    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerStarted","Data":"d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"}
Nov 03 21:45:04.250597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.250575    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerStarted","Data":"6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"}
Nov 03 21:45:04.800519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:04.800462    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" podStartSLOduration=4.800445373 podStartE2EDuration="4.800445373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:04.797266281 +0000 UTC m=+3406.428579758" watchObservedRunningTime="2026-11-03 21:45:04.800445373 +0000 UTC m=+3406.431758850"
Nov 03 21:45:05.016172 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.016124    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" podStartSLOduration=5.016112042 podStartE2EDuration="5.016112042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:05.012909536 +0000 UTC m=+3406.644223013" watchObservedRunningTime="2026-11-03 21:45:05.016112042 +0000 UTC m=+3406.647425519"
Nov 03 21:45:05.122607 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.122524    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" podStartSLOduration=5.122510234 podStartE2EDuration="5.122510234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:05.120750302 +0000 UTC m=+3406.752063779" watchObservedRunningTime="2026-11-03 21:45:05.122510234 +0000 UTC m=+3406.753823711"
Nov 03 21:45:05.261835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.261795    5533 generic.go:356] "Generic (PLEG): container finished" podID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerID="d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca" exitCode=60
Nov 03 21:45:05.262358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.261932    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerDied","Data":"d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"}
Nov 03 21:45:05.262358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.262020    5533 scope.go:122] "RemoveContainer" containerID="1e61ffe9ee8b3547402ad31cf13ec857f38868a8bd36aaa61c32681640803827"
Nov 03 21:45:05.263424 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.263400    5533 scope.go:122] "RemoveContainer" containerID="d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"
Nov 03 21:45:05.264235 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.263805    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j_odh-ai-gateway-infra(ebb9bcfa-6e46-4175-a7f4-604a19b939a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3"
Nov 03 21:45:05.265456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.265425    5533 generic.go:356] "Generic (PLEG): container finished" podID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerID="6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f" exitCode=60
Nov 03 21:45:05.265545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.265512    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerDied","Data":"6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"}
Nov 03 21:45:05.265776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.265758    5533 scope.go:122] "RemoveContainer" containerID="6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"
Nov 03 21:45:05.266072 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.266046    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-nwh7l_odh-ai-gateway-infra(2663262f-7068-4492-8c8e-0b9b8e6a4278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278"
Nov 03 21:45:05.267924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.267901    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerID="cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c" exitCode=60
Nov 03 21:45:05.268054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.267941    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerDied","Data":"cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"}
Nov 03 21:45:05.268418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.268397    5533 scope.go:122] "RemoveContainer" containerID="cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"
Nov 03 21:45:05.268819 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.268786    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k_odh-ai-gateway-infra(8d4a74c2-1cbe-4f55-b104-146db6e3095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d"
Nov 03 21:45:05.270592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.270571    5533 generic.go:356] "Generic (PLEG): container finished" podID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerID="b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67" exitCode=60
Nov 03 21:45:05.270715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.270665    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerDied","Data":"b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"}
Nov 03 21:45:05.270940 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.270924    5533 scope.go:122] "RemoveContainer" containerID="b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"
Nov 03 21:45:05.271191 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.271165    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv_odh-ai-gateway-infra(60aedfba-49cd-4f9d-af4e-5b407a2f3f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e"
Nov 03 21:45:05.272896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.272878    5533 generic.go:356] "Generic (PLEG): container finished" podID="059e5404-273a-443a-9656-387e5130f21b" containerID="3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1" exitCode=60
Nov 03 21:45:05.272996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.272945    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerDied","Data":"3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"}
Nov 03 21:45:05.273208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.273186    5533 scope.go:122] "RemoveContainer" containerID="3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"
Nov 03 21:45:05.273461 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.273437    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk_odh-ai-gateway-infra(059e5404-273a-443a-9656-387e5130f21b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" podUID="059e5404-273a-443a-9656-387e5130f21b"
Nov 03 21:45:05.274791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.274770    5533 generic.go:356] "Generic (PLEG): container finished" podID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerID="0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7" exitCode=60
Nov 03 21:45:05.274861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.274847    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerDied","Data":"0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"}
Nov 03 21:45:05.275104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.275086    5533 scope.go:122] "RemoveContainer" containerID="0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"
Nov 03 21:45:05.275356 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.275336    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt_odh-ai-gateway-infra(5d9901fd-9657-4bb1-822e-b6be43a1db2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e"
Nov 03 21:45:05.276970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.276947    5533 generic.go:356] "Generic (PLEG): container finished" podID="0419eda7-4ff1-4caf-a989-c56e34236958" containerID="2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b" exitCode=60
Nov 03 21:45:05.277251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.277223    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerDied","Data":"2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"}
Nov 03 21:45:05.277369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.277349    5533 scope.go:122] "RemoveContainer" containerID="2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"
Nov 03 21:45:05.277587 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.277566    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr_odh-ai-gateway-infra(0419eda7-4ff1-4caf-a989-c56e34236958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" podUID="0419eda7-4ff1-4caf-a989-c56e34236958"
Nov 03 21:45:05.279005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.278993    5533 scope.go:122] "RemoveContainer" containerID="6a902a9e09defdf717ce8952fb21aeb817a339763f35d9a2d3462741feb22a62"
Nov 03 21:45:05.279252 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.279230    5533 generic.go:356] "Generic (PLEG): container finished" podID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerID="52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7" exitCode=60
Nov 03 21:45:05.279345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.279294    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerDied","Data":"52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"}
Nov 03 21:45:05.279558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.279540    5533 scope.go:122] "RemoveContainer" containerID="52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"
Nov 03 21:45:05.279780 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.279761    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn_odh-ai-gateway-infra(85a2b3ff-9c0b-4ccd-869d-26a1a943b55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d"
Nov 03 21:45:05.281317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.281296    5533 generic.go:356] "Generic (PLEG): container finished" podID="158ab112-364a-4cf3-80ea-d21808276177" containerID="743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24" exitCode=60
Nov 03 21:45:05.281403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.281360    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerDied","Data":"743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"}
Nov 03 21:45:05.281608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.281592    5533 scope.go:122] "RemoveContainer" containerID="743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"
Nov 03 21:45:05.281869 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.281845    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz_odh-ai-gateway-infra(158ab112-364a-4cf3-80ea-d21808276177)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" podUID="158ab112-364a-4cf3-80ea-d21808276177"
Nov 03 21:45:05.283373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.283355    5533 generic.go:356] "Generic (PLEG): container finished" podID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerID="4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017" exitCode=60
Nov 03 21:45:05.283466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.283428    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerDied","Data":"4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"}
Nov 03 21:45:05.283650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.283622    5533 scope.go:122] "RemoveContainer" containerID="4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"
Nov 03 21:45:05.283895 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.283867    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q_odh-ai-gateway-infra(84805bc0-e9ad-49a0-9cfe-da316107c256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256"
Nov 03 21:45:05.285309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.285288    5533 generic.go:356] "Generic (PLEG): container finished" podID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerID="2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124" exitCode=60
Nov 03 21:45:05.285398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.285341    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerDied","Data":"2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"}
Nov 03 21:45:05.285533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.285517    5533 scope.go:122] "RemoveContainer" containerID="2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"
Nov 03 21:45:05.285743 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.285720    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw_odh-ai-gateway-infra(0999f1d3-6a28-4a9a-822f-981ef6dea956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956"
Nov 03 21:45:05.287204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.287184    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerID="1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97" exitCode=60
Nov 03 21:45:05.287315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.287209    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerDied","Data":"1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"}
Nov 03 21:45:05.287432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.287416    5533 scope.go:122] "RemoveContainer" containerID="1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"
Nov 03 21:45:05.287604 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:05.287584    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf_odh-ai-gateway-infra(b1e90c9c-b003-44b3-8f7e-97be647173f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6"
Nov 03 21:45:05.293304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.293283    5533 scope.go:122] "RemoveContainer" containerID="5531a9a75efc3844d27cc5c6b8fdf0614aa11828599e62c32ac5a203d6be82f2"
Nov 03 21:45:05.308505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.308472    5533 scope.go:122] "RemoveContainer" containerID="f08ee17d17397fb3c2f9c6e14d8b7476ab5937385e7b47b1ac671ed27d1f3165"
Nov 03 21:45:05.322577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.322559    5533 scope.go:122] "RemoveContainer" containerID="90d8f459a1828e74caf0fe737e08ca9ce467bc0b60cc3dde1b19d724e5c54532"
Nov 03 21:45:05.335725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.335693    5533 scope.go:122] "RemoveContainer" containerID="a6802f667952bdc1d50cf7f831794c95893b5e3814804c632d5c74b87c2b75b2"
Nov 03 21:45:05.349309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.349287    5533 scope.go:122] "RemoveContainer" containerID="cd365f2ffaed236f5bb48d184811ccf0f7ea916d8f1e81f6290b89efb4fb8db9"
Nov 03 21:45:05.362074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.362058    5533 scope.go:122] "RemoveContainer" containerID="433dc0362378d6d8b379ac96611f41b2ebc0a451129ac59492ef463ca84ee726"
Nov 03 21:45:05.374773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.374755    5533 scope.go:122] "RemoveContainer" containerID="830823b7cd840cd59cae66a1c31a7bcf14b65d443699b9197da8286d2e063603"
Nov 03 21:45:05.387723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.387706    5533 scope.go:122] "RemoveContainer" containerID="fd9b58def8de803864adf5afd700ad47d4aa658a94408b1a29cf7c3b45a991de"
Nov 03 21:45:05.400491 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.400474    5533 scope.go:122] "RemoveContainer" containerID="f23fe142ad485fdbd7bf867390f69c9a0f5fcbe191085cfee47639075756e892"
Nov 03 21:45:05.413040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.413025    5533 scope.go:122] "RemoveContainer" containerID="5f5d3e26f5d2ba1a173e817f2f1619239711c8e4b46a49e421fc4dfa643713b7"
Nov 03 21:45:05.555630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:05.555589    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" podStartSLOduration=5.555577235 podStartE2EDuration="5.555577235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:05.553291691 +0000 UTC m=+3407.184605168" watchObservedRunningTime="2026-11-03 21:45:05.555577235 +0000 UTC m=+3407.186890740"
Nov 03 21:45:06.296822 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.296792    5533 scope.go:122] "RemoveContainer" containerID="d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"
Nov 03 21:45:06.297274 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.297041    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j_odh-ai-gateway-infra(ebb9bcfa-6e46-4175-a7f4-604a19b939a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3"
Nov 03 21:45:06.298792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.298768    5533 scope.go:122] "RemoveContainer" containerID="6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"
Nov 03 21:45:06.299005 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.298981    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-nwh7l_odh-ai-gateway-infra(2663262f-7068-4492-8c8e-0b9b8e6a4278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278"
Nov 03 21:45:06.300394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.300377    5533 scope.go:122] "RemoveContainer" containerID="cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"
Nov 03 21:45:06.300604 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.300579    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k_odh-ai-gateway-infra(8d4a74c2-1cbe-4f55-b104-146db6e3095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d"
Nov 03 21:45:06.301981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.301955    5533 scope.go:122] "RemoveContainer" containerID="b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"
Nov 03 21:45:06.302185 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.302162    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv_odh-ai-gateway-infra(60aedfba-49cd-4f9d-af4e-5b407a2f3f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e"
Nov 03 21:45:06.303500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.303483    5533 scope.go:122] "RemoveContainer" containerID="3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"
Nov 03 21:45:06.303734 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.303711    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk_odh-ai-gateway-infra(059e5404-273a-443a-9656-387e5130f21b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" podUID="059e5404-273a-443a-9656-387e5130f21b"
Nov 03 21:45:06.304964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.304943    5533 scope.go:122] "RemoveContainer" containerID="0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"
Nov 03 21:45:06.305180 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.305159    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt_odh-ai-gateway-infra(5d9901fd-9657-4bb1-822e-b6be43a1db2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e"
Nov 03 21:45:06.306443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.306420    5533 scope.go:122] "RemoveContainer" containerID="2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"
Nov 03 21:45:06.306667 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.306617    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr_odh-ai-gateway-infra(0419eda7-4ff1-4caf-a989-c56e34236958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" podUID="0419eda7-4ff1-4caf-a989-c56e34236958"
Nov 03 21:45:06.308036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.308017    5533 scope.go:122] "RemoveContainer" containerID="52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"
Nov 03 21:45:06.308222 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.308197    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn_odh-ai-gateway-infra(85a2b3ff-9c0b-4ccd-869d-26a1a943b55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d"
Nov 03 21:45:06.309529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.309512    5533 scope.go:122] "RemoveContainer" containerID="743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"
Nov 03 21:45:06.309741 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.309718    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz_odh-ai-gateway-infra(158ab112-364a-4cf3-80ea-d21808276177)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" podUID="158ab112-364a-4cf3-80ea-d21808276177"
Nov 03 21:45:06.310961 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.310944    5533 scope.go:122] "RemoveContainer" containerID="4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"
Nov 03 21:45:06.311171 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.311147    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q_odh-ai-gateway-infra(84805bc0-e9ad-49a0-9cfe-da316107c256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256"
Nov 03 21:45:06.312436 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.312418    5533 scope.go:122] "RemoveContainer" containerID="2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"
Nov 03 21:45:06.312663 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.312609    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw_odh-ai-gateway-infra(0999f1d3-6a28-4a9a-822f-981ef6dea956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956"
Nov 03 21:45:06.313861 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:06.313847    5533 scope.go:122] "RemoveContainer" containerID="1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"
Nov 03 21:45:06.314012 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:06.313997    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf_odh-ai-gateway-infra(b1e90c9c-b003-44b3-8f7e-97be647173f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6"
Nov 03 21:45:15.486533 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:15.486503    5533 scope.go:122] "RemoveContainer" containerID="6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"
Nov 03 21:45:16.413143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:16.413105    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerStarted","Data":"c98c6610e4f2bf72f6aad8081f151a19546d8e9dadfe5c9fe2a67363ea8f7853"}
Nov 03 21:45:16.486685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:16.486625    5533 scope.go:122] "RemoveContainer" containerID="743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"
Nov 03 21:45:16.487104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:16.486741    5533 scope.go:122] "RemoveContainer" containerID="d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"
Nov 03 21:45:16.487104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:16.486887    5533 scope.go:122] "RemoveContainer" containerID="3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"
Nov 03 21:45:16.487104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:16.486998    5533 scope.go:122] "RemoveContainer" containerID="2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"
Nov 03 21:45:17.427790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.427751    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerStarted","Data":"f24d738147feddc59585f56e6ed47ecbe886c9e3fb43bf2ab6c173406d00d54b"}
Nov 03 21:45:17.429910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.429874    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerStarted","Data":"12dc84417a56735a67e6d76625805ff132f800ff243cbb70c987b58607ed693d"}
Nov 03 21:45:17.432129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.432099    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerStarted","Data":"be67be53ba186e51d02c493fb1ff19489e7976aa37ea63703ea8c0a990da8415"}
Nov 03 21:45:17.434121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.434098    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerStarted","Data":"c126eea46ab520f5c8da4bd870dac4ac22fe4c6f8d17345c1e2139876b29a45d"}
Nov 03 21:45:17.436082 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.436060    5533 generic.go:356] "Generic (PLEG): container finished" podID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerID="c98c6610e4f2bf72f6aad8081f151a19546d8e9dadfe5c9fe2a67363ea8f7853" exitCode=60
Nov 03 21:45:17.436199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.436117    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerDied","Data":"c98c6610e4f2bf72f6aad8081f151a19546d8e9dadfe5c9fe2a67363ea8f7853"}
Nov 03 21:45:17.436199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.436165    5533 scope.go:122] "RemoveContainer" containerID="6472f66e1a483282a2e0048556c2873efdfc734af288cd905b9a0c283d17d13f"
Nov 03 21:45:17.436472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.436450    5533 scope.go:122] "RemoveContainer" containerID="c98c6610e4f2bf72f6aad8081f151a19546d8e9dadfe5c9fe2a67363ea8f7853"
Nov 03 21:45:17.436788 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:17.436748    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-nwh7l_odh-ai-gateway-infra(2663262f-7068-4492-8c8e-0b9b8e6a4278)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278"
Nov 03 21:45:17.486723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.486693    5533 scope.go:122] "RemoveContainer" containerID="cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"
Nov 03 21:45:17.487104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.486843    5533 scope.go:122] "RemoveContainer" containerID="b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"
Nov 03 21:45:17.487104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.487007    5533 scope.go:122] "RemoveContainer" containerID="0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"
Nov 03 21:45:17.487225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.487142    5533 scope.go:122] "RemoveContainer" containerID="2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"
Nov 03 21:45:17.487295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.487254    5533 scope.go:122] "RemoveContainer" containerID="4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"
Nov 03 21:45:17.535709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:17.535678    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:18.474675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.474324    5533 generic.go:356] "Generic (PLEG): container finished" podID="158ab112-364a-4cf3-80ea-d21808276177" containerID="be67be53ba186e51d02c493fb1ff19489e7976aa37ea63703ea8c0a990da8415" exitCode=60
Nov 03 21:45:18.474675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.474409    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerDied","Data":"be67be53ba186e51d02c493fb1ff19489e7976aa37ea63703ea8c0a990da8415"}
Nov 03 21:45:18.474675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.474503    5533 scope.go:122] "RemoveContainer" containerID="743fbf8a7fa01a38feb2d922df7e906eccb78f4bba9fd56633e052462aaf7c24"
Nov 03 21:45:18.475000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.474963    5533 scope.go:122] "RemoveContainer" containerID="be67be53ba186e51d02c493fb1ff19489e7976aa37ea63703ea8c0a990da8415"
Nov 03 21:45:18.477688 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:18.475272    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz_odh-ai-gateway-infra(158ab112-364a-4cf3-80ea-d21808276177)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" podUID="158ab112-364a-4cf3-80ea-d21808276177"
Nov 03 21:45:18.484485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.480185    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerStarted","Data":"fdbb73600a8eb18ed59ea99dc9ded53334f3380858781ecde9efce962e7b6fae"}
Nov 03 21:45:18.484485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.483345    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerStarted","Data":"b0e059cd9e2f8969bec0b7f7af812b4fada9a61f3357edbdb76aa755df223089"}
Nov 03 21:45:18.491468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.490327    5533 scope.go:122] "RemoveContainer" containerID="52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"
Nov 03 21:45:18.491468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.491109    5533 scope.go:122] "RemoveContainer" containerID="1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"
Nov 03 21:45:18.494312 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.494283    5533 generic.go:356] "Generic (PLEG): container finished" podID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerID="c126eea46ab520f5c8da4bd870dac4ac22fe4c6f8d17345c1e2139876b29a45d" exitCode=60
Nov 03 21:45:18.504433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.504405    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerDied","Data":"c126eea46ab520f5c8da4bd870dac4ac22fe4c6f8d17345c1e2139876b29a45d"}
Nov 03 21:45:18.504738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.504721    5533 scope.go:122] "RemoveContainer" containerID="c126eea46ab520f5c8da4bd870dac4ac22fe4c6f8d17345c1e2139876b29a45d"
Nov 03 21:45:18.504945 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:18.504926    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j_odh-ai-gateway-infra(ebb9bcfa-6e46-4175-a7f4-604a19b939a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3"
Nov 03 21:45:18.508446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.508421    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerStarted","Data":"acb467daaf52152731724f513c6ff8a02ead7b5377260c054a606b01e9003203"}
Nov 03 21:45:18.510844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.510814    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerStarted","Data":"9a4cb849bb030d5f1d5d7977936cf619f9f464c602c5a6fa50eb2b0707064eaf"}
Nov 03 21:45:18.513485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.513464    5533 generic.go:356] "Generic (PLEG): container finished" podID="059e5404-273a-443a-9656-387e5130f21b" containerID="f24d738147feddc59585f56e6ed47ecbe886c9e3fb43bf2ab6c173406d00d54b" exitCode=60
Nov 03 21:45:18.513593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.513523    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerDied","Data":"f24d738147feddc59585f56e6ed47ecbe886c9e3fb43bf2ab6c173406d00d54b"}
Nov 03 21:45:18.513846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.513828    5533 scope.go:122] "RemoveContainer" containerID="f24d738147feddc59585f56e6ed47ecbe886c9e3fb43bf2ab6c173406d00d54b"
Nov 03 21:45:18.514035 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:18.514014    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk_odh-ai-gateway-infra(059e5404-273a-443a-9656-387e5130f21b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" podUID="059e5404-273a-443a-9656-387e5130f21b"
Nov 03 21:45:18.515226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.515210    5533 scope.go:122] "RemoveContainer" containerID="d89ded0a39e392fb62f834f4087fdcf9a258310da7c45207d0119ebe37be1eca"
Nov 03 21:45:18.515693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.515671    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerStarted","Data":"39943a9a6b1b5eb2a52c9e5dc4923931f070e4c01b55d7f28a2cb1da7fd2c377"}
Nov 03 21:45:18.518563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.518541    5533 generic.go:356] "Generic (PLEG): container finished" podID="0419eda7-4ff1-4caf-a989-c56e34236958" containerID="12dc84417a56735a67e6d76625805ff132f800ff243cbb70c987b58607ed693d" exitCode=60
Nov 03 21:45:18.518668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.518616    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerDied","Data":"12dc84417a56735a67e6d76625805ff132f800ff243cbb70c987b58607ed693d"}
Nov 03 21:45:18.518951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.518931    5533 scope.go:122] "RemoveContainer" containerID="12dc84417a56735a67e6d76625805ff132f800ff243cbb70c987b58607ed693d"
Nov 03 21:45:18.519196 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:18.519173    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr_odh-ai-gateway-infra(0419eda7-4ff1-4caf-a989-c56e34236958)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" podUID="0419eda7-4ff1-4caf-a989-c56e34236958"
Nov 03 21:45:18.555721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.555600    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:18.584057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.583712    5533 scope.go:122] "RemoveContainer" containerID="3ce8ca8a32bab27d09baa08ad4a1e3f9c5ddcca2bf6081beccd3a956b9090be1"
Nov 03 21:45:18.646015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.631166    5533 scope.go:122] "RemoveContainer" containerID="2338979bd402bf0061f61e455a744cc832fdf58a3bf998e0a016d838a4c97c9b"
Nov 03 21:45:18.703376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.703289    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:18.774934 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.774905    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:18.999617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:18.999582    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:19.094814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.094789    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:19.192295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.192270    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\") pod \"2663262f-7068-4492-8c8e-0b9b8e6a4278\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") "
Nov 03 21:45:19.192445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.192429    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\" (UniqueName: \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\") pod \"2663262f-7068-4492-8c8e-0b9b8e6a4278\" (UID: \"2663262f-7068-4492-8c8e-0b9b8e6a4278\") "
Nov 03 21:45:19.192694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.192669    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca" pod "2663262f-7068-4492-8c8e-0b9b8e6a4278" (UID: "2663262f-7068-4492-8c8e-0b9b8e6a4278"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.192917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.192899    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2663262f-7068-4492-8c8e-0b9b8e6a4278-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.194510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.194478    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm" pod "2663262f-7068-4492-8c8e-0b9b8e6a4278" (UID: "2663262f-7068-4492-8c8e-0b9b8e6a4278"). InnerVolumeSpecName "kube-api-access-mjxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.263427 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.263346    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:45:19.294236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.294201    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjxbm\" (UniqueName: \"kubernetes.io/projected/2663262f-7068-4492-8c8e-0b9b8e6a4278-kube-api-access-mjxbm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.529113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.529041    5533 generic.go:356] "Generic (PLEG): container finished" podID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerID="9a4cb849bb030d5f1d5d7977936cf619f9f464c602c5a6fa50eb2b0707064eaf" exitCode=60
Nov 03 21:45:19.529523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.529109    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerDied","Data":"9a4cb849bb030d5f1d5d7977936cf619f9f464c602c5a6fa50eb2b0707064eaf"}
Nov 03 21:45:19.529523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.529164    5533 scope.go:122] "RemoveContainer" containerID="b7e3686e437e89aeeffd7aeac62bbd1a643de3db43424b93df98194070a25a67"
Nov 03 21:45:19.529523 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.529494    5533 scope.go:122] "RemoveContainer" containerID="9a4cb849bb030d5f1d5d7977936cf619f9f464c602c5a6fa50eb2b0707064eaf"
Nov 03 21:45:19.529840 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.529814    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv_odh-ai-gateway-infra(60aedfba-49cd-4f9d-af4e-5b407a2f3f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e"
Nov 03 21:45:19.533013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.532993    5533 generic.go:356] "Generic (PLEG): container finished" podID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerID="39943a9a6b1b5eb2a52c9e5dc4923931f070e4c01b55d7f28a2cb1da7fd2c377" exitCode=60
Nov 03 21:45:19.533112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.533062    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerDied","Data":"39943a9a6b1b5eb2a52c9e5dc4923931f070e4c01b55d7f28a2cb1da7fd2c377"}
Nov 03 21:45:19.533423 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.533402    5533 scope.go:122] "RemoveContainer" containerID="39943a9a6b1b5eb2a52c9e5dc4923931f070e4c01b55d7f28a2cb1da7fd2c377"
Nov 03 21:45:19.533708 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.533687    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt_odh-ai-gateway-infra(5d9901fd-9657-4bb1-822e-b6be43a1db2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e"
Nov 03 21:45:19.536957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.536931    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerStarted","Data":"1d401d62e3df7faba31039e974411d7d4541469fea674b863ec88e47cd14df7a"}
Nov 03 21:45:19.541799 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.541775    5533 generic.go:356] "Generic (PLEG): container finished" podID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerID="fdbb73600a8eb18ed59ea99dc9ded53334f3380858781ecde9efce962e7b6fae" exitCode=60
Nov 03 21:45:19.541889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.541859    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerDied","Data":"fdbb73600a8eb18ed59ea99dc9ded53334f3380858781ecde9efce962e7b6fae"}
Nov 03 21:45:19.542174 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.542157    5533 scope.go:122] "RemoveContainer" containerID="fdbb73600a8eb18ed59ea99dc9ded53334f3380858781ecde9efce962e7b6fae"
Nov 03 21:45:19.542431 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.542409    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q_odh-ai-gateway-infra(84805bc0-e9ad-49a0-9cfe-da316107c256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256"
Nov 03 21:45:19.546811 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.546786    5533 scope.go:122] "RemoveContainer" containerID="0f50eba3adb5d9d62e5d3e1b798054337c79e150fed57e7f0d311b7fdca43eb7"
Nov 03 21:45:19.548714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.548687    5533 generic.go:356] "Generic (PLEG): container finished" podID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerID="b0e059cd9e2f8969bec0b7f7af812b4fada9a61f3357edbdb76aa755df223089" exitCode=60
Nov 03 21:45:19.548826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.548780    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerDied","Data":"b0e059cd9e2f8969bec0b7f7af812b4fada9a61f3357edbdb76aa755df223089"}
Nov 03 21:45:19.549064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.549046    5533 scope.go:122] "RemoveContainer" containerID="b0e059cd9e2f8969bec0b7f7af812b4fada9a61f3357edbdb76aa755df223089"
Nov 03 21:45:19.549277 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.549255    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw_odh-ai-gateway-infra(0999f1d3-6a28-4a9a-822f-981ef6dea956)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956"
Nov 03 21:45:19.552804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.552220    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerStarted","Data":"c17a4d33ff6fcb2464c8ae56f6d0fa16fb70283f3221aeb158d0228b8645ed7f"}
Nov 03 21:45:19.561678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.558407    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l" event={"ID":"2663262f-7068-4492-8c8e-0b9b8e6a4278","Type":"ContainerDied","Data":"986fbc3f82b685954331d3a1a3da2620041f4a0cd6a405a9b815836cf58611a2"}
Nov 03 21:45:19.561678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.558511    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"
Nov 03 21:45:19.574336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.574302    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerID="acb467daaf52152731724f513c6ff8a02ead7b5377260c054a606b01e9003203" exitCode=60
Nov 03 21:45:19.574467 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.574368    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerDied","Data":"acb467daaf52152731724f513c6ff8a02ead7b5377260c054a606b01e9003203"}
Nov 03 21:45:19.574818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.574793    5533 scope.go:122] "RemoveContainer" containerID="acb467daaf52152731724f513c6ff8a02ead7b5377260c054a606b01e9003203"
Nov 03 21:45:19.575141 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:19.575118    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k_odh-ai-gateway-infra(8d4a74c2-1cbe-4f55-b104-146db6e3095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d"
Nov 03 21:45:19.580506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.580489    5533 scope.go:122] "RemoveContainer" containerID="4af8eb2a8628fe2047257ead4747f4f2874f1c0e4df2addf2e6c26cc5d2ad017"
Nov 03 21:45:19.602321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.602296    5533 scope.go:122] "RemoveContainer" containerID="2e3b7ded25d6d287e9fb63a24fccf2028aeac9924be3ea35cfdd7646be672124"
Nov 03 21:45:19.629933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.628855    5533 scope.go:122] "RemoveContainer" containerID="c98c6610e4f2bf72f6aad8081f151a19546d8e9dadfe5c9fe2a67363ea8f7853"
Nov 03 21:45:19.660040 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.657885    5533 scope.go:122] "RemoveContainer" containerID="cfe121b78ceb40595903e1bc6063b1bce4fd5e8c84567dd81e9dd0e8aa6ab44c"
Nov 03 21:45:19.725662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.725609    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:19.803573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.803487    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:19.813215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.813189    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:19.823519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.823490    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:19.828830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.828806    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:19.903237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903205    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\") pod \"158ab112-364a-4cf3-80ea-d21808276177\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") "
Nov 03 21:45:19.903392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903376    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\" (UniqueName: \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\") pod \"0419eda7-4ff1-4caf-a989-c56e34236958\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") "
Nov 03 21:45:19.903468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903452    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\") pod \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") "
Nov 03 21:45:19.903534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903484    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\" (UniqueName: \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\") pod \"158ab112-364a-4cf3-80ea-d21808276177\" (UID: \"158ab112-364a-4cf3-80ea-d21808276177\") "
Nov 03 21:45:19.903597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903545    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\" (UniqueName: \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\") pod \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\" (UID: \"ebb9bcfa-6e46-4175-a7f4-604a19b939a3\") "
Nov 03 21:45:19.903597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903561    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca" pod "158ab112-364a-4cf3-80ea-d21808276177" (UID: "158ab112-364a-4cf3-80ea-d21808276177"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.903597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903571    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\") pod \"059e5404-273a-443a-9656-387e5130f21b\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") "
Nov 03 21:45:19.903769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903678    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\") pod \"0419eda7-4ff1-4caf-a989-c56e34236958\" (UID: \"0419eda7-4ff1-4caf-a989-c56e34236958\") "
Nov 03 21:45:19.903769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903716    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\" (UniqueName: \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\") pod \"059e5404-273a-443a-9656-387e5130f21b\" (UID: \"059e5404-273a-443a-9656-387e5130f21b\") "
Nov 03 21:45:19.903917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.903897    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca" pod "059e5404-273a-443a-9656-387e5130f21b" (UID: "059e5404-273a-443a-9656-387e5130f21b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.904272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.904242    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca" pod "ebb9bcfa-6e46-4175-a7f4-604a19b939a3" (UID: "ebb9bcfa-6e46-4175-a7f4-604a19b939a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.904517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.904482    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158ab112-364a-4cf3-80ea-d21808276177-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.904517 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.904512    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.904770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.904536    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059e5404-273a-443a-9656-387e5130f21b-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.904914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.904892    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca" pod "0419eda7-4ff1-4caf-a989-c56e34236958" (UID: "0419eda7-4ff1-4caf-a989-c56e34236958"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.907776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.905966    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2" pod "0419eda7-4ff1-4caf-a989-c56e34236958" (UID: "0419eda7-4ff1-4caf-a989-c56e34236958"). InnerVolumeSpecName "kube-api-access-s6pc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.907776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.906349    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx" pod "ebb9bcfa-6e46-4175-a7f4-604a19b939a3" (UID: "ebb9bcfa-6e46-4175-a7f4-604a19b939a3"). InnerVolumeSpecName "kube-api-access-kpkhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.907776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.906751    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl" pod "059e5404-273a-443a-9656-387e5130f21b" (UID: "059e5404-273a-443a-9656-387e5130f21b"). InnerVolumeSpecName "kube-api-access-7lhdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.907776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.906954    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69" pod "158ab112-364a-4cf3-80ea-d21808276177" (UID: "158ab112-364a-4cf3-80ea-d21808276177"). InnerVolumeSpecName "kube-api-access-5mw69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.948440 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:19.948412    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:20.006113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.006076    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6pc2\" (UniqueName: \"kubernetes.io/projected/0419eda7-4ff1-4caf-a989-c56e34236958-kube-api-access-s6pc2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.006113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.006114    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mw69\" (UniqueName: \"kubernetes.io/projected/158ab112-364a-4cf3-80ea-d21808276177-kube-api-access-5mw69\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.006323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.006130    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpkhx\" (UniqueName: \"kubernetes.io/projected/ebb9bcfa-6e46-4175-a7f4-604a19b939a3-kube-api-access-kpkhx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.006323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.006151    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0419eda7-4ff1-4caf-a989-c56e34236958-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.006323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.006166    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhdl\" (UniqueName: \"kubernetes.io/projected/059e5404-273a-443a-9656-387e5130f21b-kube-api-access-7lhdl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.166479 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.166370    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:20.341561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.340546    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:20.346713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.346667    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-nwh7l"]
Nov 03 21:45:20.385005 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.384959    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:20.497303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.497263    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" path="/var/lib/kubelet/pods/2663262f-7068-4492-8c8e-0b9b8e6a4278/volumes"
Nov 03 21:45:20.601072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.597466    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk" event={"ID":"059e5404-273a-443a-9656-387e5130f21b","Type":"ContainerDied","Data":"3cc7b571ddcaefb3130e8d4abfc43727629eb9e06ad570ca22ecdd8895ac66aa"}
Nov 03 21:45:20.601072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.597536    5533 scope.go:122] "RemoveContainer" containerID="f24d738147feddc59585f56e6ed47ecbe886c9e3fb43bf2ab6c173406d00d54b"
Nov 03 21:45:20.601072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.597705    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"
Nov 03 21:45:20.618581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.618062    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:20.620288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.619088    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr" event={"ID":"0419eda7-4ff1-4caf-a989-c56e34236958","Type":"ContainerDied","Data":"85a2000224eb6f018809337f63f2b86b6687a1cfc0ced8743f89ee687fe45385"}
Nov 03 21:45:20.620288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.619120    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"
Nov 03 21:45:20.639138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.632576    5533 generic.go:356] "Generic (PLEG): container finished" podID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerID="1d401d62e3df7faba31039e974411d7d4541469fea674b863ec88e47cd14df7a" exitCode=60
Nov 03 21:45:20.639138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.632678    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerDied","Data":"1d401d62e3df7faba31039e974411d7d4541469fea674b863ec88e47cd14df7a"}
Nov 03 21:45:20.639138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.633201    5533 scope.go:122] "RemoveContainer" containerID="1d401d62e3df7faba31039e974411d7d4541469fea674b863ec88e47cd14df7a"
Nov 03 21:45:20.639138 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:20.633563    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn_odh-ai-gateway-infra(85a2b3ff-9c0b-4ccd-869d-26a1a943b55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d"
Nov 03 21:45:20.641664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.641427    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz" event={"ID":"158ab112-364a-4cf3-80ea-d21808276177","Type":"ContainerDied","Data":"893a48b3245de68b8b2a93e441713ba66d72bcce63d42023f92f8207a8c2efaf"}
Nov 03 21:45:20.641664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.641554    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.690565    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerID="c17a4d33ff6fcb2464c8ae56f6d0fa16fb70283f3221aeb158d0228b8645ed7f" exitCode=60
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.690713    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerDied","Data":"c17a4d33ff6fcb2464c8ae56f6d0fa16fb70283f3221aeb158d0228b8645ed7f"}
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.690830    5533 scope.go:122] "RemoveContainer" containerID="12dc84417a56735a67e6d76625805ff132f800ff243cbb70c987b58607ed693d"
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.691474    5533 scope.go:122] "RemoveContainer" containerID="c17a4d33ff6fcb2464c8ae56f6d0fa16fb70283f3221aeb158d0228b8645ed7f"
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:20.691768    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf_odh-ai-gateway-infra(b1e90c9c-b003-44b3-8f7e-97be647173f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6"
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.693220    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j" event={"ID":"ebb9bcfa-6e46-4175-a7f4-604a19b939a3","Type":"ContainerDied","Data":"95f92da24bbf450fe998b2092fbb2497ea8944b2bc719ec60cc320d067495e6d"}
Nov 03 21:45:20.698028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.693295    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"
Nov 03 21:45:20.777945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.777817    5533 scope.go:122] "RemoveContainer" containerID="52697f8f37544932e1439efcb6d1703e039c4243ae3ac77c227fc3cfd52ce5e7"
Nov 03 21:45:20.807950 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.807845    5533 scope.go:122] "RemoveContainer" containerID="be67be53ba186e51d02c493fb1ff19489e7976aa37ea63703ea8c0a990da8415"
Nov 03 21:45:20.825709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.824666    5533 scope.go:122] "RemoveContainer" containerID="1e20392f0e9bdd8d5cda3915fa077a66e012ec73f77879be6222be8a28953e97"
Nov 03 21:45:20.889036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.887987    5533 scope.go:122] "RemoveContainer" containerID="2bcb0ad916049e3102d406f8bd052a1770cb720cabe1f7e704f0df0abc61f0da"
Nov 03 21:45:20.914346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.914273    5533 scope.go:122] "RemoveContainer" containerID="c126eea46ab520f5c8da4bd870dac4ac22fe4c6f8d17345c1e2139876b29a45d"
Nov 03 21:45:20.914688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.914508    5533 scope.go:122] "RemoveContainer" containerID="a6ce77d63f34704d7f31ee37c4efd43565f99e8f5f568c45e8b6988676d78896"
Nov 03 21:45:20.915397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.915378    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:20.918764 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.918741    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:20.938725 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.938699    5533 scope.go:122] "RemoveContainer" containerID="e5d3a8349a0aa19e4f9cc604c814b043dfee7e13d8f94a47245f80323cc8ae95"
Nov 03 21:45:20.972502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.972478    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:20.976833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:20.976813    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:21.004407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.004335    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:21.030399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.030372    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\") pod \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") "
Nov 03 21:45:21.030501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.030487    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\") pod \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") "
Nov 03 21:45:21.030676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.030656    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\" (UniqueName: \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\") pod \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\" (UID: \"8d4a74c2-1cbe-4f55-b104-146db6e3095d\") "
Nov 03 21:45:21.030817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.030791    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\" (UniqueName: \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\") pod \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\" (UID: \"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e\") "
Nov 03 21:45:21.030895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.030813    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca" pod "8d4a74c2-1cbe-4f55-b104-146db6e3095d" (UID: "8d4a74c2-1cbe-4f55-b104-146db6e3095d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:21.031033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.031013    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca" pod "60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" (UID: "60aedfba-49cd-4f9d-af4e-5b407a2f3f7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:21.031422 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.031396    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4a74c2-1cbe-4f55-b104-146db6e3095d-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.031507 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.031422    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.033015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.032993    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl" pod "8d4a74c2-1cbe-4f55-b104-146db6e3095d" (UID: "8d4a74c2-1cbe-4f55-b104-146db6e3095d"). InnerVolumeSpecName "kube-api-access-4glpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:21.033164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.033138    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt" pod "60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" (UID: "60aedfba-49cd-4f9d-af4e-5b407a2f3f7e"). InnerVolumeSpecName "kube-api-access-cq5qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:21.072351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.072326    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:21.132529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132501    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\" (UniqueName: \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\") pod \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") "
Nov 03 21:45:21.132671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132606    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\") pod \"0999f1d3-6a28-4a9a-822f-981ef6dea956\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") "
Nov 03 21:45:21.132671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132662    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\" (UniqueName: \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\") pod \"0999f1d3-6a28-4a9a-822f-981ef6dea956\" (UID: \"0999f1d3-6a28-4a9a-822f-981ef6dea956\") "
Nov 03 21:45:21.132810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132701    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\" (UniqueName: \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\") pod \"84805bc0-e9ad-49a0-9cfe-da316107c256\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") "
Nov 03 21:45:21.132868 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132811    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\") pod \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\" (UID: \"5d9901fd-9657-4bb1-822e-b6be43a1db2e\") "
Nov 03 21:45:21.132938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.132891    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\") pod \"84805bc0-e9ad-49a0-9cfe-da316107c256\" (UID: \"84805bc0-e9ad-49a0-9cfe-da316107c256\") "
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133329    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca" pod "5d9901fd-9657-4bb1-822e-b6be43a1db2e" (UID: "5d9901fd-9657-4bb1-822e-b6be43a1db2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133474    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca" pod "0999f1d3-6a28-4a9a-822f-981ef6dea956" (UID: "0999f1d3-6a28-4a9a-822f-981ef6dea956"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133499    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca" pod "84805bc0-e9ad-49a0-9cfe-da316107c256" (UID: "84805bc0-e9ad-49a0-9cfe-da316107c256"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133549    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4glpl\" (UniqueName: \"kubernetes.io/projected/8d4a74c2-1cbe-4f55-b104-146db6e3095d-kube-api-access-4glpl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133572    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5qt\" (UniqueName: \"kubernetes.io/projected/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e-kube-api-access-cq5qt\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.133610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.133588    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9901fd-9657-4bb1-822e-b6be43a1db2e-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.135619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.135598    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd" pod "5d9901fd-9657-4bb1-822e-b6be43a1db2e" (UID: "5d9901fd-9657-4bb1-822e-b6be43a1db2e"). InnerVolumeSpecName "kube-api-access-577nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:21.135728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.135657    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc" pod "0999f1d3-6a28-4a9a-822f-981ef6dea956" (UID: "0999f1d3-6a28-4a9a-822f-981ef6dea956"). InnerVolumeSpecName "kube-api-access-5qvkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:21.135728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.135701    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m" pod "84805bc0-e9ad-49a0-9cfe-da316107c256" (UID: "84805bc0-e9ad-49a0-9cfe-da316107c256"). InnerVolumeSpecName "kube-api-access-zxw5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:21.179304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.179259    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:21.187036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.187010    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4d1799-29895705-csrrz"]
Nov 03 21:45:21.235212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.235183    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577nd\" (UniqueName: \"kubernetes.io/projected/5d9901fd-9657-4bb1-822e-b6be43a1db2e-kube-api-access-577nd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.235212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.235213    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0999f1d3-6a28-4a9a-822f-981ef6dea956-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.235365 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.235229    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvkc\" (UniqueName: \"kubernetes.io/projected/0999f1d3-6a28-4a9a-822f-981ef6dea956-kube-api-access-5qvkc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.235365 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.235247    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxw5m\" (UniqueName: \"kubernetes.io/projected/84805bc0-e9ad-49a0-9cfe-da316107c256-kube-api-access-zxw5m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.235365 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.235259    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84805bc0-e9ad-49a0-9cfe-da316107c256-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:21.348786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.348703    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:21.352844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.352816    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895705-9kvzr"]
Nov 03 21:45:21.573830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.573797    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:21.631329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.631228    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:21.637787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.637759    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895705-nbz9j"]
Nov 03 21:45:21.704778 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.704748    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"
Nov 03 21:45:21.704952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.704779    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k" event={"ID":"8d4a74c2-1cbe-4f55-b104-146db6e3095d","Type":"ContainerDied","Data":"ead6d6f8af24b12f601a7b67d204a4aef4a02b2d34ad1c00eda2e3f5388afd85"}
Nov 03 21:45:21.704952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.704864    5533 scope.go:122] "RemoveContainer" containerID="acb467daaf52152731724f513c6ff8a02ead7b5377260c054a606b01e9003203"
Nov 03 21:45:21.706623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.706588    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv" event={"ID":"60aedfba-49cd-4f9d-af4e-5b407a2f3f7e","Type":"ContainerDied","Data":"fb8099e8275df4c73d4a2f0ddf8441fe0db3a806d22d3f8cf05286db254c660e"}
Nov 03 21:45:21.706744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.706734    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"
Nov 03 21:45:21.714381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.714359    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"
Nov 03 21:45:21.714512 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.714360    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt" event={"ID":"5d9901fd-9657-4bb1-822e-b6be43a1db2e","Type":"ContainerDied","Data":"5fc44c7c4db9c7c734c527bb704ccad780ce677ea01d60f2abf17ce6566ce1c0"}
Nov 03 21:45:21.719189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.719169    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q" event={"ID":"84805bc0-e9ad-49a0-9cfe-da316107c256","Type":"ContainerDied","Data":"38b4ca92fb7a2837bec75a275594ac059cd4adc1d5722d8b4992962af1e833b1"}
Nov 03 21:45:21.719278 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.719186    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"
Nov 03 21:45:21.720685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.720660    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw" event={"ID":"0999f1d3-6a28-4a9a-822f-981ef6dea956","Type":"ContainerDied","Data":"bf5c3c3456c25a6f17b345a576dab1678734859b5ea6938639b62f14a00124ab"}
Nov 03 21:45:21.720794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.720674    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"
Nov 03 21:45:21.733696 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.733679    5533 scope.go:122] "RemoveContainer" containerID="9a4cb849bb030d5f1d5d7977936cf619f9f464c602c5a6fa50eb2b0707064eaf"
Nov 03 21:45:21.759855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.759828    5533 scope.go:122] "RemoveContainer" containerID="39943a9a6b1b5eb2a52c9e5dc4923931f070e4c01b55d7f28a2cb1da7fd2c377"
Nov 03 21:45:21.781579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.781553    5533 scope.go:122] "RemoveContainer" containerID="fdbb73600a8eb18ed59ea99dc9ded53334f3380858781ecde9efce962e7b6fae"
Nov 03 21:45:21.797798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.796750    5533 scope.go:122] "RemoveContainer" containerID="b0e059cd9e2f8969bec0b7f7af812b4fada9a61f3357edbdb76aa755df223089"
Nov 03 21:45:21.925927 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.925900    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:21.974665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:21.972621    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:22.025741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.025707    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:22.035881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.035854    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0fc5f-29895705-p9pbk"]
Nov 03 21:45:22.048380 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.048352    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\") pod \"b1e90c9c-b003-44b3-8f7e-97be647173f6\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") "
Nov 03 21:45:22.048527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.048513    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\" (UniqueName: \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\") pod \"b1e90c9c-b003-44b3-8f7e-97be647173f6\" (UID: \"b1e90c9c-b003-44b3-8f7e-97be647173f6\") "
Nov 03 21:45:22.048781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.048757    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca" pod "b1e90c9c-b003-44b3-8f7e-97be647173f6" (UID: "b1e90c9c-b003-44b3-8f7e-97be647173f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:22.049048 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.049029    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e90c9c-b003-44b3-8f7e-97be647173f6-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:22.050676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.050655    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9" pod "b1e90c9c-b003-44b3-8f7e-97be647173f6" (UID: "b1e90c9c-b003-44b3-8f7e-97be647173f6"). InnerVolumeSpecName "kube-api-access-kslg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:22.149760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.149700    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\") pod \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") "
Nov 03 21:45:22.149760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.149746    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\" (UniqueName: \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\") pod \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\" (UID: \"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d\") "
Nov 03 21:45:22.150070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.150050    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca" pod "85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" (UID: "85a2b3ff-9c0b-4ccd-869d-26a1a943b55d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:22.150335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.150318    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kslg9\" (UniqueName: \"kubernetes.io/projected/b1e90c9c-b003-44b3-8f7e-97be647173f6-kube-api-access-kslg9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:22.150372 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.150341    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:22.151972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.151944    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp" pod "85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" (UID: "85a2b3ff-9c0b-4ccd-869d-26a1a943b55d"). InnerVolumeSpecName "kube-api-access-7rhpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:22.251982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.251947    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rhpp\" (UniqueName: \"kubernetes.io/projected/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d-kube-api-access-7rhpp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:22.495196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.495158    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" path="/var/lib/kubelet/pods/0419eda7-4ff1-4caf-a989-c56e34236958/volumes"
Nov 03 21:45:22.495630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.495614    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059e5404-273a-443a-9656-387e5130f21b" path="/var/lib/kubelet/pods/059e5404-273a-443a-9656-387e5130f21b/volumes"
Nov 03 21:45:22.496031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.496016    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158ab112-364a-4cf3-80ea-d21808276177" path="/var/lib/kubelet/pods/158ab112-364a-4cf3-80ea-d21808276177/volumes"
Nov 03 21:45:22.496736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.496715    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" path="/var/lib/kubelet/pods/ebb9bcfa-6e46-4175-a7f4-604a19b939a3/volumes"
Nov 03 21:45:22.581496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.581281    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:22.585130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.585102    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895705-sqnzt"]
Nov 03 21:45:22.733862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.733831    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"
Nov 03 21:45:22.734262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.733832    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn" event={"ID":"85a2b3ff-9c0b-4ccd-869d-26a1a943b55d","Type":"ContainerDied","Data":"062b7d5236020d05a04a2994a257bca248270286db0c68edad47841210e6e0f6"}
Nov 03 21:45:22.734262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.733979    5533 scope.go:122] "RemoveContainer" containerID="1d401d62e3df7faba31039e974411d7d4541469fea674b863ec88e47cd14df7a"
Nov 03 21:45:22.738358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.738333    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"
Nov 03 21:45:22.738485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.738361    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf" event={"ID":"b1e90c9c-b003-44b3-8f7e-97be647173f6","Type":"ContainerDied","Data":"aa0c7d631810a53d7b642fca06f0a6bc296ccacd95f90a1392c14171b5f745de"}
Nov 03 21:45:22.747297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.747279    5533 scope.go:122] "RemoveContainer" containerID="c17a4d33ff6fcb2464c8ae56f6d0fa16fb70283f3221aeb158d0228b8645ed7f"
Nov 03 21:45:22.799104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.799078    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:22.802447 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.802420    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895705-kz5qw"]
Nov 03 21:45:22.903807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.903781    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:22.909935 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:22.909912    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895705-tpvvv"]
Nov 03 21:45:23.121298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.121215    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:23.125495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.125468    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895705-r6j5k"]
Nov 03 21:45:23.227287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.227255    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:23.232072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.232049    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c70fca-29895705-pfx4q"]
Nov 03 21:45:23.334339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.334310    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:23.338065 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.338042    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895705-ltnvf"]
Nov 03 21:45:23.440452 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.440422    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:23.445234 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:23.445205    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895705-5w6bn"]
Nov 03 21:45:24.497877 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.497844    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" path="/var/lib/kubelet/pods/0999f1d3-6a28-4a9a-822f-981ef6dea956/volumes"
Nov 03 21:45:24.498351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.498257    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" path="/var/lib/kubelet/pods/5d9901fd-9657-4bb1-822e-b6be43a1db2e/volumes"
Nov 03 21:45:24.498660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.498622    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" path="/var/lib/kubelet/pods/60aedfba-49cd-4f9d-af4e-5b407a2f3f7e/volumes"
Nov 03 21:45:24.499396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.499370    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" path="/var/lib/kubelet/pods/84805bc0-e9ad-49a0-9cfe-da316107c256/volumes"
Nov 03 21:45:24.499854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.499834    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" path="/var/lib/kubelet/pods/85a2b3ff-9c0b-4ccd-869d-26a1a943b55d/volumes"
Nov 03 21:45:24.500313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.500294    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" path="/var/lib/kubelet/pods/8d4a74c2-1cbe-4f55-b104-146db6e3095d/volumes"
Nov 03 21:45:24.501073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:24.501053    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" path="/var/lib/kubelet/pods/b1e90c9c-b003-44b3-8f7e-97be647173f6/volumes"
Nov 03 21:45:27.143624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:27.143587    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:45:37.932818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.932787    5533 generic.go:356] "Generic (PLEG): container finished" podID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerID="ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b" exitCode=0
Nov 03 21:45:37.933244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.932858    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerDied","Data":"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"}
Nov 03 21:45:37.933289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.933250    5533 scope.go:122] "RemoveContainer" containerID="ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"
Nov 03 21:45:37.934798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.934772    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:45:37.934867 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.934837    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:45:37.934913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:37.934870    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:45:38.942165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:38.942129    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerStarted","Data":"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"}
Nov 03 21:45:38.990096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:38.990054    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:45:39.992628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:39.992587    5533 patch_prober.go:29] interesting pod/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 21:45:39.993124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:39.992678    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 21:45:40.993715 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:40.993679    5533 patch_prober.go:29] interesting pod/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 21:45:40.994299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:40.994266    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 21:45:41.381714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:41.380726    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:45:41.991130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:41.991072    5533 patch_prober.go:29] interesting pod/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 21:45:41.991361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:41.991205    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 21:45:42.990135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:42.990099    5533 patch_prober.go:29] interesting pod/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 21:45:42.990532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:42.990166    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 21:45:43.991117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:43.991083    5533 patch_prober.go:29] interesting pod/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 21:45:43.991535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:43.991147    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 21:45:44.996247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:44.996213    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:45:44.997445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:44.997422    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:45:46.751743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:46.751704    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:45:46.857108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:46.857077    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:45:49.261031 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:49.260999    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:45:51.752590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:51.752529    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" podUID="4e215a7c-aba4-401b-a776-ac2b30561e1c" containerName="payload-processing" containerID="cri-o://d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727" gracePeriod=25
Nov 03 21:45:51.858174 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:51.858126    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" podUID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" containerName="payload-pre-processing" containerID="cri-o://f215bf1dcfd95f2cf6534bb372da0e56cc4a14f382266f0a663904f892b32f1f" gracePeriod=25
Nov 03 21:45:52.036991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.036962    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:45:52.085538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.085508    5533 generic.go:356] "Generic (PLEG): container finished" podID="4e215a7c-aba4-401b-a776-ac2b30561e1c" containerID="d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727" exitCode=0
Nov 03 21:45:52.085744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.085564    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" event={"ID":"4e215a7c-aba4-401b-a776-ac2b30561e1c","Type":"ContainerDied","Data":"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"}
Nov 03 21:45:52.085744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.085619    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb" event={"ID":"4e215a7c-aba4-401b-a776-ac2b30561e1c","Type":"ContainerDied","Data":"ed3be5a43516161eab81d1b797eabb66e0981912d457a00b1697ba6493082ad2"}
Nov 03 21:45:52.085744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.085660    5533 scope.go:122] "RemoveContainer" containerID="d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"
Nov 03 21:45:52.085744 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.085568    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"
Nov 03 21:45:52.089230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.089204    5533 generic.go:356] "Generic (PLEG): container finished" podID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" containerID="f215bf1dcfd95f2cf6534bb372da0e56cc4a14f382266f0a663904f892b32f1f" exitCode=0
Nov 03 21:45:52.089401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.089382    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" event={"ID":"98a1ad7a-81eb-4c32-acb7-2f332900be1a","Type":"ContainerDied","Data":"f215bf1dcfd95f2cf6534bb372da0e56cc4a14f382266f0a663904f892b32f1f"}
Nov 03 21:45:52.101306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.101229    5533 scope.go:122] "RemoveContainer" containerID="d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"
Nov 03 21:45:52.101540 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:45:52.101504    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727\": container with ID starting with d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727 not found: ID does not exist" containerID="d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"
Nov 03 21:45:52.101620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.101542    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727"} err="failed to get container status \"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727\": rpc error: code = NotFound desc = could not find container \"d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727\": container with ID starting with d1a1a6197c85753f97d4ae77c493c8c49a5300b936888b45358456fd25c38727 not found: ID does not exist"
Nov 03 21:45:52.161573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.161548    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:45:52.188842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.188793    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\") pod \"4e215a7c-aba4-401b-a776-ac2b30561e1c\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") "
Nov 03 21:45:52.189710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.189067    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\" (UniqueName: \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\") pod \"4e215a7c-aba4-401b-a776-ac2b30561e1c\" (UID: \"4e215a7c-aba4-401b-a776-ac2b30561e1c\") "
Nov 03 21:45:52.189710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.189443    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume" pod "4e215a7c-aba4-401b-a776-ac2b30561e1c" (UID: "4e215a7c-aba4-401b-a776-ac2b30561e1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:52.189710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.189624    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e215a7c-aba4-401b-a776-ac2b30561e1c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:52.191581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.191556    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f" pod "4e215a7c-aba4-401b-a776-ac2b30561e1c" (UID: "4e215a7c-aba4-401b-a776-ac2b30561e1c"). InnerVolumeSpecName "kube-api-access-7vq7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:52.290481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.290424    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\") pod \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") "
Nov 03 21:45:52.290481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.290473    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\" (UniqueName: \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\") pod \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\" (UID: \"98a1ad7a-81eb-4c32-acb7-2f332900be1a\") "
Nov 03 21:45:52.290892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.290870    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume" pod "98a1ad7a-81eb-4c32-acb7-2f332900be1a" (UID: "98a1ad7a-81eb-4c32-acb7-2f332900be1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:52.290951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.290926    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vq7f\" (UniqueName: \"kubernetes.io/projected/4e215a7c-aba4-401b-a776-ac2b30561e1c-kube-api-access-7vq7f\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:52.292566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.292546    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm" pod "98a1ad7a-81eb-4c32-acb7-2f332900be1a" (UID: "98a1ad7a-81eb-4c32-acb7-2f332900be1a"). InnerVolumeSpecName "kube-api-access-vv4tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:52.391731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.391701    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1ad7a-81eb-4c32-acb7-2f332900be1a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:52.391731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.391728    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4tm\" (UniqueName: \"kubernetes.io/projected/98a1ad7a-81eb-4c32-acb7-2f332900be1a-kube-api-access-vv4tm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:52.506335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.506298    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:45:52.509944 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:52.509918    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0fc5f-d79fbb7cd-5m7sb"]
Nov 03 21:45:53.099157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:53.099079    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v" event={"ID":"98a1ad7a-81eb-4c32-acb7-2f332900be1a","Type":"ContainerDied","Data":"a71e9c69ffdbb52a04f1133c33a8a3d090838aa6277998a6bd9bf0d80641aa78"}
Nov 03 21:45:53.099157 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:53.099142    5533 scope.go:122] "RemoveContainer" containerID="f215bf1dcfd95f2cf6534bb372da0e56cc4a14f382266f0a663904f892b32f1f"
Nov 03 21:45:53.099616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:53.099093    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"
Nov 03 21:45:53.256954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:53.256924    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:45:53.303406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:53.303377    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0fc5f-756697d847-t6p6v"]
Nov 03 21:45:54.494906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:54.494873    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e215a7c-aba4-401b-a776-ac2b30561e1c" path="/var/lib/kubelet/pods/4e215a7c-aba4-401b-a776-ac2b30561e1c/volumes"
Nov 03 21:45:54.495318 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:45:54.495299    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" path="/var/lib/kubelet/pods/98a1ad7a-81eb-4c32-acb7-2f332900be1a/volumes"
Nov 03 21:46:03.290682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:03.290596    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:46:13.068716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.068681    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:46:13.069381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069356    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.069381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069377    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.069381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069388    5533 cpu_manager.go:414] "containerMap: removing container" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069393    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069404    5533 cpu_manager.go:414] "containerMap: removing container" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069411    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069420    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069424    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069431    5533 cpu_manager.go:414] "containerMap: removing container" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069436    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069445    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4e215a7c-aba4-401b-a776-ac2b30561e1c" containerName="payload-processing"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069450    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e215a7c-aba4-401b-a776-ac2b30561e1c" containerName="payload-processing"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069456    5533 cpu_manager.go:414] "containerMap: removing container" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069461    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069466    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069470    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069478    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069483    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069488    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069493    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069503    5533 cpu_manager.go:414] "containerMap: removing container" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069508    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069515    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069520    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069535    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069539    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069545    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069549    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069555    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069560    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069567    5533 cpu_manager.go:414] "containerMap: removing container" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069571    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069582    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069586    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069591    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069596    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069602    5533 cpu_manager.go:414] "containerMap: removing container" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069606    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069613    5533 cpu_manager.go:414] "containerMap: removing container" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069618    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069629    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069645    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069652    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069656    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069662    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069667    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069676    5533 cpu_manager.go:414] "containerMap: removing container" podUID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" containerName="payload-pre-processing"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069681    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" containerName="payload-pre-processing"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069687    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069692    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069701    5533 cpu_manager.go:414] "containerMap: removing container" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.069673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069705    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069715    5533 cpu_manager.go:414] "containerMap: removing container" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069720    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069727    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069731    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069739    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069745    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069844    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069853    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069858    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069863    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069869    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069875    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069884    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069889    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a1ad7a-81eb-4c32-acb7-2f332900be1a" containerName="payload-pre-processing"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069895    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069900    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069906    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069912    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069919    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069924    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069930    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069934    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069940    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069946    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069950    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e215a7c-aba4-401b-a776-ac2b30561e1c" containerName="payload-processing"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069955    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069959    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069964    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069970    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069975    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069982    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb9bcfa-6e46-4175-a7f4-604a19b939a3" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069987    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.069994    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.070001    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:46:13.072264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.070006    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:46:13.075326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.075306    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249697    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249737    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249754    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249781    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249822    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.249992 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.249924    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qr7\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.250215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.250046    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.250215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.250115    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.250215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.250149    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.294548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.294501    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:46:13.350995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.350920    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.350995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.350959    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.350995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.350976    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.350996    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351021    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351067    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qr7\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351144    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351178    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351215    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351411    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351713    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.351936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351908    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.352115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.351989    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.352251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.352098    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.353892 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.353868    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.354001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.353904    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.408755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.408727    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.409162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.409137    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qr7\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\") pod \"e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") " pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:13.691222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:13.691154    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:14.111685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.111578    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:46:14.144811 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:46:14.144783    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81284fa4_4933_46ee_9295_767af2657392.slice/crio-6afd2730ed8430438667c4a5411594614dc7b6e4bdb098fd04e20443bfd269a7 WatchSource:0}: Error finding container 6afd2730ed8430438667c4a5411594614dc7b6e4bdb098fd04e20443bfd269a7: Status 404 returned error can't find the container with id 6afd2730ed8430438667c4a5411594614dc7b6e4bdb098fd04e20443bfd269a7
Nov 03 21:46:14.147375 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.147335    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.147497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.147421    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.147497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.147466    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.269184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.269146    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" event={"ID":"81284fa4-4933-46ee-9295-767af2657392","Type":"ContainerStarted","Data":"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"}
Nov 03 21:46:14.269464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.269439    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" event={"ID":"81284fa4-4933-46ee-9295-767af2657392","Type":"ContainerStarted","Data":"6afd2730ed8430438667c4a5411594614dc7b6e4bdb098fd04e20443bfd269a7"}
Nov 03 21:46:14.272004 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.271979    5533 generic.go:356] "Generic (PLEG): container finished" podID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerID="164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29" exitCode=0
Nov 03 21:46:14.272126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.272089    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerDied","Data":"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"}
Nov 03 21:46:14.272448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.272430    5533 scope.go:122] "RemoveContainer" containerID="164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"
Nov 03 21:46:14.274080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.274050    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.274235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.274134    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.274235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.274188    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:46:14.387954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.387867    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podStartSLOduration=1.387852 podStartE2EDuration="1.387852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:14.385157789 +0000 UTC m=+3476.016471266" watchObservedRunningTime="2026-11-03 21:46:14.387852 +0000 UTC m=+3476.019165477"
Nov 03 21:46:14.411363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.411332    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:46:14.692133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:14.692096    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:15.302661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:15.302384    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerStarted","Data":"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"}
Nov 03 21:46:15.696659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:15.696551    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:46:15.696659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:15.696614    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 21:46:16.096713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:16.096075    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:46:16.099658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:16.098015    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 03 21:46:16.099658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:16.098066    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 03 21:46:16.692656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:16.692033    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:46:16.692656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:16.692112    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 21:46:17.095714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:17.095034    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 03 21:46:17.095714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:17.095106    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 03 21:46:17.694534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:17.694062    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:46:17.694534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:17.694130    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 21:46:18.104710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:18.104018    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 03 21:46:18.104710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:18.104082    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 03 21:46:18.692387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:18.691937    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:46:18.692387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:18.691997    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 21:46:19.098713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:19.098027    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 03 21:46:19.098713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:19.098101    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 03 21:46:19.322174 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:46:19.322133    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:46:19.704660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:19.702039    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:46:19.704660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:19.702110    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 21:46:20.102722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:20.097017    5533 patch_prober.go:29] interesting pod/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 03 21:46:20.102722 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:20.097077    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 03 21:46:20.697064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:20.697033    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:20.697548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:20.697525    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:20.698227 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:20.698209    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:46:21.040617 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:21.040534    5533 scope.go:122] "RemoveContainer" containerID="8b1e7e61a4c4ac13f2e66f7182f5fdb1b28ab07d267274f309e904c988f091b0"
Nov 03 21:46:21.050853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:21.050831    5533 scope.go:122] "RemoveContainer" containerID="a28040c7ffdd96c0e76c079ea604b7626d43ba7ba3a8fc786d8808e7fef4e1ef"
Nov 03 21:46:21.100293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:21.100267    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:46:21.100714 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:21.100693    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:46:21.101484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:21.101434    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:46:25.048066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:25.048023    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"]
Nov 03 21:46:25.094630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:25.094591    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d43e84-lbsh6"]
Nov 03 21:46:26.494143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:26.494110    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff937b78-c65b-40af-b11a-5702356e9cd2" path="/var/lib/kubelet/pods/ff937b78-c65b-40af-b11a-5702356e9cd2/volumes"
Nov 03 21:46:36.050406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:36.050361    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"]
Nov 03 21:46:36.093458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:36.093428    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1208cb28-f5rtc"]
Nov 03 21:46:36.494317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:36.494285    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702c15e1-e9ec-4683-abb7-8b9ef2cf1bff" path="/var/lib/kubelet/pods/702c15e1-e9ec-4683-abb7-8b9ef2cf1bff/volumes"
Nov 03 21:46:46.498075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:46.498041    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:46:46.498730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:46.498221    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" containerID="cri-o://b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64" gracePeriod=30
Nov 03 21:46:49.262383 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:46:49.262340    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:46:49.579615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:49.579542    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:46:49.579791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:49.579609    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:46:50.764652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:50.764591    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:46:51.757792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.757767    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:46:51.790975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.790914    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.790982    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791041    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791160    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791228    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791301    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791356    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791386    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791431    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791460    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791494    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\") pod \"168a1c45-db66-4eb3-8327-978472d5538d\" (UID: \"168a1c45-db66-4eb3-8327-978472d5538d\") "
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791517    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:51.791734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.791689    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:51.792074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.792044    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:51.792147 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.792128    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.792212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.792153    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.792212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.792171    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.792212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.792188    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.793526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.793501    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:51.793698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.793661    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "kube-api-access-s2bqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:51.793798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.793775    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:46:51.794001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.793978    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy" pod "168a1c45-db66-4eb3-8327-978472d5538d" (UID: "168a1c45-db66-4eb3-8327-978472d5538d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:51.893754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.893719    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a1c45-db66-4eb3-8327-978472d5538d-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.893905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.893758    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bqq\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-kube-api-access-s2bqq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.893905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.893775    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a1c45-db66-4eb3-8327-978472d5538d-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.893905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.893789    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a1c45-db66-4eb3-8327-978472d5538d-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:51.893905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:51.893802    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a1c45-db66-4eb3-8327-978472d5538d-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:52.647891 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.647851    5533 generic.go:356] "Generic (PLEG): container finished" podID="168a1c45-db66-4eb3-8327-978472d5538d" containerID="b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64" exitCode=0
Nov 03 21:46:52.648108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.648027    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerDied","Data":"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"}
Nov 03 21:46:52.648108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.648093    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb" event={"ID":"168a1c45-db66-4eb3-8327-978472d5538d","Type":"ContainerDied","Data":"5b3a21007cfca814408b2fad6e68a878ec286a15076ad7285545af713f0a7ef4"}
Nov 03 21:46:52.648238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.648137    5533 scope.go:122] "RemoveContainer" containerID="b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"
Nov 03 21:46:52.648367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.648352    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"
Nov 03 21:46:52.674990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.674957    5533 scope.go:122] "RemoveContainer" containerID="ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"
Nov 03 21:46:52.686439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.686418    5533 scope.go:122] "RemoveContainer" containerID="b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"
Nov 03 21:46:52.686762 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:46:52.686738    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64\": container with ID starting with b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64 not found: ID does not exist" containerID="b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"
Nov 03 21:46:52.686873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.686768    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64"} err="failed to get container status \"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64\": rpc error: code = NotFound desc = could not find container \"b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64\": container with ID starting with b780dedd05af23216bf6961945aaf6a03dae60d02ab0b2c83c498429ed7d6b64 not found: ID does not exist"
Nov 03 21:46:52.686873 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.686787    5533 scope.go:122] "RemoveContainer" containerID="ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"
Nov 03 21:46:52.687086 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:46:52.687069    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc\": container with ID starting with ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc not found: ID does not exist" containerID="ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"
Nov 03 21:46:52.687134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.687091    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc"} err="failed to get container status \"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc\": rpc error: code = NotFound desc = could not find container \"ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc\": container with ID starting with ed6fb2d2d6e42d1785ed10d13a09849d0fb71b65820f1dd7ed8c9fee8f2b00dc not found: ID does not exist"
Nov 03 21:46:52.814537 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.814465    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:46:52.860449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:52.860420    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-130a8c-openshift-default-6b58cc4cd9-s45rb"]
Nov 03 21:46:54.494955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:54.494927    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168a1c45-db66-4eb3-8327-978472d5538d" path="/var/lib/kubelet/pods/168a1c45-db66-4eb3-8327-978472d5538d/volumes"
Nov 03 21:46:55.628412 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:55.628381    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:46:55.735618 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:55.735590    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:46:55.765680 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:55.765628    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" podUID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" containerName="payload-processing" containerID="cri-o://0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015" gracePeriod=25
Nov 03 21:46:56.072515 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.070747    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:46:56.184346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.184313    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\" (UniqueName: \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\") pod \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") "
Nov 03 21:46:56.184599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.184577    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\") pod \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\" (UID: \"6e5b88d9-de6f-449e-a4ea-50db7739b1c7\") "
Nov 03 21:46:56.185089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.185054    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume" pod "6e5b88d9-de6f-449e-a4ea-50db7739b1c7" (UID: "6e5b88d9-de6f-449e-a4ea-50db7739b1c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:56.185232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.185181    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:56.187054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.187025    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx" pod "6e5b88d9-de6f-449e-a4ea-50db7739b1c7" (UID: "6e5b88d9-de6f-449e-a4ea-50db7739b1c7"). InnerVolumeSpecName "kube-api-access-94mbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:56.286907 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.286870    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mbx\" (UniqueName: \"kubernetes.io/projected/6e5b88d9-de6f-449e-a4ea-50db7739b1c7-kube-api-access-94mbx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:46:56.682502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.682460    5533 generic.go:356] "Generic (PLEG): container finished" podID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" containerID="0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015" exitCode=0
Nov 03 21:46:56.682502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.682497    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"
Nov 03 21:46:56.683041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.682514    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" event={"ID":"6e5b88d9-de6f-449e-a4ea-50db7739b1c7","Type":"ContainerDied","Data":"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"}
Nov 03 21:46:56.683041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.682574    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88" event={"ID":"6e5b88d9-de6f-449e-a4ea-50db7739b1c7","Type":"ContainerDied","Data":"2b225fdac44d0fd67277671ae35f41a115b08c4a9678f89dbb118aaf530c9e9b"}
Nov 03 21:46:56.683041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.682600    5533 scope.go:122] "RemoveContainer" containerID="0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"
Nov 03 21:46:56.694249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.694230    5533 scope.go:122] "RemoveContainer" containerID="0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"
Nov 03 21:46:56.694525 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:46:56.694505    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015\": container with ID starting with 0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015 not found: ID does not exist" containerID="0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"
Nov 03 21:46:56.694604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.694534    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015"} err="failed to get container status \"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015\": rpc error: code = NotFound desc = could not find container \"0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015\": container with ID starting with 0a0294e284a5596bd28bc9868442a940eef99cb1eadf9dfbaa8de3bf3f68b015 not found: ID does not exist"
Nov 03 21:46:56.838257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.838174    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:46:56.844820 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:56.844790    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a3dc77-fcdb464d6-ntp88"]
Nov 03 21:46:58.293675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:58.293642    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:46:58.394996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:58.394965    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:46:58.498147 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:46:58.498113    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" path="/var/lib/kubelet/pods/6e5b88d9-de6f-449e-a4ea-50db7739b1c7/volumes"
Nov 03 21:47:00.628735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.628645    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" podUID="6d487f7e-c3af-4363-869f-1fa3a43220d4" containerName="payload-processing" containerID="cri-o://69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2" gracePeriod=25
Nov 03 21:47:00.736284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.736252    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" podUID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" containerName="payload-pre-processing" containerID="cri-o://3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9" gracePeriod=25
Nov 03 21:47:00.911326 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.911302    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:47:00.944791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.944765    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\") pod \"6d487f7e-c3af-4363-869f-1fa3a43220d4\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") "
Nov 03 21:47:00.944918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.944815    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\" (UniqueName: \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\") pod \"6d487f7e-c3af-4363-869f-1fa3a43220d4\" (UID: \"6d487f7e-c3af-4363-869f-1fa3a43220d4\") "
Nov 03 21:47:00.945204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.945181    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume" pod "6d487f7e-c3af-4363-869f-1fa3a43220d4" (UID: "6d487f7e-c3af-4363-869f-1fa3a43220d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:00.945337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.945321    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d487f7e-c3af-4363-869f-1fa3a43220d4-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:00.947711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.947682    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7" pod "6d487f7e-c3af-4363-869f-1fa3a43220d4" (UID: "6d487f7e-c3af-4363-869f-1fa3a43220d4"). InnerVolumeSpecName "kube-api-access-kx7c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:00.992006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:00.991980    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:47:01.046391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.046360    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\" (UniqueName: \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\") pod \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") "
Nov 03 21:47:01.046548 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.046465    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\") pod \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\" (UID: \"c27685cb-4b7a-4e40-ad60-2fa4069eb673\") "
Nov 03 21:47:01.046855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.046833    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume" pod "c27685cb-4b7a-4e40-ad60-2fa4069eb673" (UID: "c27685cb-4b7a-4e40-ad60-2fa4069eb673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:01.047131 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.047113    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c27685cb-4b7a-4e40-ad60-2fa4069eb673-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:01.047221 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.047133    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7c7\" (UniqueName: \"kubernetes.io/projected/6d487f7e-c3af-4363-869f-1fa3a43220d4-kube-api-access-kx7c7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:01.048484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.048463    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg" pod "c27685cb-4b7a-4e40-ad60-2fa4069eb673" (UID: "c27685cb-4b7a-4e40-ad60-2fa4069eb673"). InnerVolumeSpecName "kube-api-access-5qmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:01.148183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.148126    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmpg\" (UniqueName: \"kubernetes.io/projected/c27685cb-4b7a-4e40-ad60-2fa4069eb673-kube-api-access-5qmpg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:01.724678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.724630    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d487f7e-c3af-4363-869f-1fa3a43220d4" containerID="69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2" exitCode=0
Nov 03 21:47:01.725105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.724679    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" event={"ID":"6d487f7e-c3af-4363-869f-1fa3a43220d4","Type":"ContainerDied","Data":"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"}
Nov 03 21:47:01.725105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.724703    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"
Nov 03 21:47:01.725105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.724749    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g" event={"ID":"6d487f7e-c3af-4363-869f-1fa3a43220d4","Type":"ContainerDied","Data":"75af3f190716476d9e5fe3249694e382e421279fcf4f3c0de467bdb17caa543d"}
Nov 03 21:47:01.725105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.724782    5533 scope.go:122] "RemoveContainer" containerID="69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"
Nov 03 21:47:01.726364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.726336    5533 generic.go:356] "Generic (PLEG): container finished" podID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" containerID="3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9" exitCode=0
Nov 03 21:47:01.726472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.726369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" event={"ID":"c27685cb-4b7a-4e40-ad60-2fa4069eb673","Type":"ContainerDied","Data":"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"}
Nov 03 21:47:01.726472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.726388    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"
Nov 03 21:47:01.726472 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.726415    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm" event={"ID":"c27685cb-4b7a-4e40-ad60-2fa4069eb673","Type":"ContainerDied","Data":"932f6f433e99ca5ac8e7cb53aa7745e131f9eab967061f6e71f20bb98c469c3f"}
Nov 03 21:47:01.736857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.736800    5533 scope.go:122] "RemoveContainer" containerID="69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"
Nov 03 21:47:01.737102 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:01.737078    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2\": container with ID starting with 69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2 not found: ID does not exist" containerID="69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"
Nov 03 21:47:01.737183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.737111    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2"} err="failed to get container status \"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2\": rpc error: code = NotFound desc = could not find container \"69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2\": container with ID starting with 69ada1f7a4fed32c91d0d69ec1450b981a5a15810ed64e3b1bf24227c49434b2 not found: ID does not exist"
Nov 03 21:47:01.737183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.737145    5533 scope.go:122] "RemoveContainer" containerID="3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"
Nov 03 21:47:01.746263 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.746245    5533 scope.go:122] "RemoveContainer" containerID="3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"
Nov 03 21:47:01.746517 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:01.746496    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9\": container with ID starting with 3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9 not found: ID does not exist" containerID="3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"
Nov 03 21:47:01.746590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.746524    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9"} err="failed to get container status \"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9\": rpc error: code = NotFound desc = could not find container \"3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9\": container with ID starting with 3e2d8adff57d573f2e9ae7ef1acc56e12bca2ad8e1dcc42601eb44f1e5af33b9 not found: ID does not exist"
Nov 03 21:47:01.882319 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.882292    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:47:01.885593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.885567    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7da0f7-556ddf9d95-smlsm"]
Nov 03 21:47:01.989666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.989628    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:47:01.996897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:01.996869    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7da0f7-64b978f5db-mrd5g"]
Nov 03 21:47:02.494526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:02.494497    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d487f7e-c3af-4363-869f-1fa3a43220d4" path="/var/lib/kubelet/pods/6d487f7e-c3af-4363-869f-1fa3a43220d4/volumes"
Nov 03 21:47:02.494915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:02.494900    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" path="/var/lib/kubelet/pods/c27685cb-4b7a-4e40-ad60-2fa4069eb673/volumes"
Nov 03 21:47:03.294805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.294757    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" podUID="9c649afa-7c53-4c91-a325-72260ba6484c" containerName="payload-processing" containerID="cri-o://27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647" gracePeriod=25
Nov 03 21:47:03.396007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.395966    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" podUID="147306bc-00da-47c2-9e2f-ca6b566485f4" containerName="payload-pre-processing" containerID="cri-o://934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb" gracePeriod=25
Nov 03 21:47:03.585988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.585952    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:47:03.678173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.678142    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\" (UniqueName: \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\") pod \"9c649afa-7c53-4c91-a325-72260ba6484c\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") "
Nov 03 21:47:03.678331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.678234    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\") pod \"9c649afa-7c53-4c91-a325-72260ba6484c\" (UID: \"9c649afa-7c53-4c91-a325-72260ba6484c\") "
Nov 03 21:47:03.678749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.678721    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume" pod "9c649afa-7c53-4c91-a325-72260ba6484c" (UID: "9c649afa-7c53-4c91-a325-72260ba6484c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:03.678876 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.678820    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c649afa-7c53-4c91-a325-72260ba6484c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:03.680327 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.680303    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh" pod "9c649afa-7c53-4c91-a325-72260ba6484c" (UID: "9c649afa-7c53-4c91-a325-72260ba6484c"). InnerVolumeSpecName "kube-api-access-sccwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:03.688211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.688186    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:47:03.745428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.745401    5533 generic.go:356] "Generic (PLEG): container finished" podID="9c649afa-7c53-4c91-a325-72260ba6484c" containerID="27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647" exitCode=0
Nov 03 21:47:03.745554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.745445    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"
Nov 03 21:47:03.745554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.745482    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" event={"ID":"9c649afa-7c53-4c91-a325-72260ba6484c","Type":"ContainerDied","Data":"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"}
Nov 03 21:47:03.745673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.745569    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr" event={"ID":"9c649afa-7c53-4c91-a325-72260ba6484c","Type":"ContainerDied","Data":"1dada78fd989489b4f8963bd56f4dc98ec461f61db25192f52b8d5cf3f2b1815"}
Nov 03 21:47:03.745673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.745597    5533 scope.go:122] "RemoveContainer" containerID="27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"
Nov 03 21:47:03.747117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.747097    5533 generic.go:356] "Generic (PLEG): container finished" podID="147306bc-00da-47c2-9e2f-ca6b566485f4" containerID="934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb" exitCode=0
Nov 03 21:47:03.747248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.747119    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" event={"ID":"147306bc-00da-47c2-9e2f-ca6b566485f4","Type":"ContainerDied","Data":"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"}
Nov 03 21:47:03.747248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.747139    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"
Nov 03 21:47:03.747248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.747164    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv" event={"ID":"147306bc-00da-47c2-9e2f-ca6b566485f4","Type":"ContainerDied","Data":"0dcf7363481cb3afd89e5aa62ce9730229600d1573dcc49b55afe87c17d86efc"}
Nov 03 21:47:03.756367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.756352    5533 scope.go:122] "RemoveContainer" containerID="27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"
Nov 03 21:47:03.756623 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:03.756605    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647\": container with ID starting with 27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647 not found: ID does not exist" containerID="27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"
Nov 03 21:47:03.756736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.756648    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647"} err="failed to get container status \"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647\": rpc error: code = NotFound desc = could not find container \"27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647\": container with ID starting with 27848d9d5dd2ef0dc5b63a25287060c9364d7233f13173f659fe7b3649e94647 not found: ID does not exist"
Nov 03 21:47:03.756736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.756673    5533 scope.go:122] "RemoveContainer" containerID="934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"
Nov 03 21:47:03.767584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.767563    5533 scope.go:122] "RemoveContainer" containerID="934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"
Nov 03 21:47:03.767920 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:03.767898    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb\": container with ID starting with 934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb not found: ID does not exist" containerID="934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"
Nov 03 21:47:03.768015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.767924    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb"} err="failed to get container status \"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb\": rpc error: code = NotFound desc = could not find container \"934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb\": container with ID starting with 934e7937ed787bf155e8a197ef3450bbe014c052c458c9856292e618db74d4eb not found: ID does not exist"
Nov 03 21:47:03.779481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.779460    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\" (UniqueName: \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\") pod \"147306bc-00da-47c2-9e2f-ca6b566485f4\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") "
Nov 03 21:47:03.779673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.779658    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\") pod \"147306bc-00da-47c2-9e2f-ca6b566485f4\" (UID: \"147306bc-00da-47c2-9e2f-ca6b566485f4\") "
Nov 03 21:47:03.779983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.779964    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sccwh\" (UniqueName: \"kubernetes.io/projected/9c649afa-7c53-4c91-a325-72260ba6484c-kube-api-access-sccwh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:03.780123 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.780100    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume" pod "147306bc-00da-47c2-9e2f-ca6b566485f4" (UID: "147306bc-00da-47c2-9e2f-ca6b566485f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:03.781475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.781452    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c" pod "147306bc-00da-47c2-9e2f-ca6b566485f4" (UID: "147306bc-00da-47c2-9e2f-ca6b566485f4"). InnerVolumeSpecName "kube-api-access-6km8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:03.881042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.880976    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147306bc-00da-47c2-9e2f-ca6b566485f4-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:03.881042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.881005    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6km8c\" (UniqueName: \"kubernetes.io/projected/147306bc-00da-47c2-9e2f-ca6b566485f4-kube-api-access-6km8c\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:03.961078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.961051    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:47:03.964612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:03.964589    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5e2db-7ff7fd9ccd-qnkfr"]
Nov 03 21:47:04.163267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:04.163197    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:47:04.168991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:04.168965    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e2db-ffcc69499-k2chv"]
Nov 03 21:47:04.494866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:04.494833    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147306bc-00da-47c2-9e2f-ca6b566485f4" path="/var/lib/kubelet/pods/147306bc-00da-47c2-9e2f-ca6b566485f4/volumes"
Nov 03 21:47:04.495346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:04.495249    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c649afa-7c53-4c91-a325-72260ba6484c" path="/var/lib/kubelet/pods/9c649afa-7c53-4c91-a325-72260ba6484c/volumes"
Nov 03 21:47:06.522990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:06.522962    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:47:07.858539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:07.858503    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:47:07.859109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:07.858712    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy" containerID="cri-o://bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5" gracePeriod=30
Nov 03 21:47:07.873814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:07.873788    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:47:07.873994 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:07.873976    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy" containerID="cri-o://d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e" gracePeriod=30
Nov 03 21:47:10.360269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.360232    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"]
Nov 03 21:47:10.363986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.363951    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9c649afa-7c53-4c91-a325-72260ba6484c" containerName="payload-processing"
Nov 03 21:47:10.363986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.363984    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c649afa-7c53-4c91-a325-72260ba6484c" containerName="payload-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364019    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364028    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364039    5533 cpu_manager.go:414] "containerMap: removing container" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364046    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364057    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364064    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364075    5533 cpu_manager.go:414] "containerMap: removing container" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364082    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364096    5533 cpu_manager.go:414] "containerMap: removing container" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364104    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364114    5533 cpu_manager.go:414] "containerMap: removing container" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364122    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364133    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364141    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9901fd-9657-4bb1-822e-b6be43a1db2e" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364157    5533 cpu_manager.go:414] "containerMap: removing container" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364166    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059e5404-273a-443a-9656-387e5130f21b" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364190    5533 cpu_manager.go:414] "containerMap: removing container" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364197    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364208    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d487f7e-c3af-4363-869f-1fa3a43220d4" containerName="payload-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364215    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d487f7e-c3af-4363-869f-1fa3a43220d4" containerName="payload-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364228    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" containerName="payload-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364235    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" containerName="payload-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364248    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" containerName="payload-pre-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364255    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" containerName="payload-pre-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364265    5533 cpu_manager.go:414] "containerMap: removing container" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364272    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364289    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364296    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364309    5533 cpu_manager.go:414] "containerMap: removing container" podUID="147306bc-00da-47c2-9e2f-ca6b566485f4" containerName="payload-pre-processing"
Nov 03 21:47:10.364307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364317    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147306bc-00da-47c2-9e2f-ca6b566485f4" containerName="payload-pre-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364467    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e90c9c-b003-44b3-8f7e-97be647173f6" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364479    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="84805bc0-e9ad-49a0-9cfe-da316107c256" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364489    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364498    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2663262f-7068-4492-8c8e-0b9b8e6a4278" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364512    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0999f1d3-6a28-4a9a-822f-981ef6dea956" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364522    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="168a1c45-db66-4eb3-8327-978472d5538d" containerName="istio-proxy"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364533    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d487f7e-c3af-4363-869f-1fa3a43220d4" containerName="payload-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364541    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5b88d9-de6f-449e-a4ea-50db7739b1c7" containerName="payload-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364550    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27685cb-4b7a-4e40-ad60-2fa4069eb673" containerName="payload-pre-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364562    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a2b3ff-9c0b-4ccd-869d-26a1a943b55d" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364572    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="147306bc-00da-47c2-9e2f-ca6b566485f4" containerName="payload-pre-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364584    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0419eda7-4ff1-4caf-a989-c56e34236958" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364592    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4a74c2-1cbe-4f55-b104-146db6e3095d" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364603    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aedfba-49cd-4f9d-af4e-5b407a2f3f7e" containerName="cleanup"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364611    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c649afa-7c53-4c91-a325-72260ba6484c" containerName="payload-processing"
Nov 03 21:47:10.366176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.364622    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="158ab112-364a-4cf3-80ea-d21808276177" containerName="cleanup"
Nov 03 21:47:10.368536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.368514    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.420656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.419829    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"]
Nov 03 21:47:10.460006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.459969    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:47:10.460176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.460016    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:47:10.464663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.464600    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.464844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.464740    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjjh\" (UniqueName: \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.464933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.464909    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.510208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.510183    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:47:10.514346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.514329    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.563955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.563923    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-267053-openshift-default-dockercfg-hpxf5\""
Nov 03 21:47:10.565612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.565570    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:47:10.566096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566069    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.566256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566116    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566190    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566287    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566316    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjjh\" (UniqueName: \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.566553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566480    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh4t\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566514    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566553 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566541    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.566788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566603    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.566862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.566807    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.567962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.567240    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.567962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.567681    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.567962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.567722    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.570309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.570245    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.624753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.624695    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjjh\" (UniqueName: \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\") pod \"maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.669314 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669289    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669359    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh4t\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669378    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669395    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669532    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669625 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669606    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.669781 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669673    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670054    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670058    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.669899    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670094    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670191    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670278    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.670816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.670534    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.672117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.672088    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.672225 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.672126    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.720217 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.720198    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:10.727169 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.727145    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.727339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.727321    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh4t\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\") pod \"e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") " pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:10.828534 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:10.828502    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:11.108712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.108675    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"]
Nov 03 21:47:11.109079 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:47:11.109061    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701a6078_2cc7_40fd_815f_ed647d10e8b0.slice/crio-7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5 WatchSource:0}: Error finding container 7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5: Status 404 returned error can't find the container with id 7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5
Nov 03 21:47:11.218496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.218460    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:47:11.218620 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:47:11.218599    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87039730_f061_4333_a820_7ba0f7de70da.slice/crio-7548c37539f57621593533d03764150c9ad9e3e458840d0e2177abe4b8fb4b17 WatchSource:0}: Error finding container 7548c37539f57621593533d03764150c9ad9e3e458840d0e2177abe4b8fb4b17: Status 404 returned error can't find the container with id 7548c37539f57621593533d03764150c9ad9e3e458840d0e2177abe4b8fb4b17
Nov 03 21:47:11.221497 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.221465    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:47:11.221585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.221524    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:47:11.221585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.221552    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:47:11.830598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.830558    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerStarted","Data":"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"}
Nov 03 21:47:11.831097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.830623    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerStarted","Data":"7548c37539f57621593533d03764150c9ad9e3e458840d0e2177abe4b8fb4b17"}
Nov 03 21:47:11.832366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.832335    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9" event={"ID":"701a6078-2cc7-40fd-815f-ed647d10e8b0","Type":"ContainerStarted","Data":"45132dd3879588ac97363540ba0c13903ed8b1a4f850acc553d54838cee0c611"}
Nov 03 21:47:11.832502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.832392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9" event={"ID":"701a6078-2cc7-40fd-815f-ed647d10e8b0","Type":"ContainerStarted","Data":"7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5"}
Nov 03 21:47:11.946186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:11.946123    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podStartSLOduration=1.9461049209999999 podStartE2EDuration="1.946104921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:11.94397312 +0000 UTC m=+3533.575286597" watchObservedRunningTime="2026-11-03 21:47:11.946104921 +0000 UTC m=+3533.577418399"
Nov 03 21:47:12.054762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.054718    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9" podStartSLOduration=2.054703703 podStartE2EDuration="2.054703703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:12.052297514 +0000 UTC m=+3533.683610991" watchObservedRunningTime="2026-11-03 21:47:12.054703703 +0000 UTC m=+3533.686017216"
Nov 03 21:47:12.107385 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:12.107305    5533 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701a6078_2cc7_40fd_815f_ed647d10e8b0.slice/crio-conmon-45132dd3879588ac97363540ba0c13903ed8b1a4f850acc553d54838cee0c611.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:47:12.829698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.829662    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:12.835223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.835194    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:12.841378 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.841356    5533 generic.go:356] "Generic (PLEG): container finished" podID="701a6078-2cc7-40fd-815f-ed647d10e8b0" containerID="45132dd3879588ac97363540ba0c13903ed8b1a4f850acc553d54838cee0c611" exitCode=0
Nov 03 21:47:12.841527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.841427    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9" event={"ID":"701a6078-2cc7-40fd-815f-ed647d10e8b0","Type":"ContainerDied","Data":"45132dd3879588ac97363540ba0c13903ed8b1a4f850acc553d54838cee0c611"}
Nov 03 21:47:12.841773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.841757    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:12.842363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:12.842344    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:47:13.173676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.167776    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:47:13.203145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.203112    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:47:13.310129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310092    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310134    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.310377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310198    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310230    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.310377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310265    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310306    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.310377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310338    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310379    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310438    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310528    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.310675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310569    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.310675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310627    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.310942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310918    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.311011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310955    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.311011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311003    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.311160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311055    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\") pod \"1999725a-e166-4784-b3c0-60d11a2bc0fc\" (UID: \"1999725a-e166-4784-b3c0-60d11a2bc0fc\") "
Nov 03 21:47:13.311160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311111    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.311281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311171    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\") pod \"81284fa4-4933-46ee-9295-767af2657392\" (UID: \"81284fa4-4933-46ee-9295-767af2657392\") "
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310331    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310460    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310668    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310692    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310826    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.310888    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311026    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.311214    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.312042    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.312257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.312218    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:13.313108 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.313084    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "kube-api-access-j7qr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:13.313247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.313225    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "kube-api-access-cgrdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:13.313812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.313785    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.313909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.313837    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:47:13.313909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.313870    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:13.314707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.314685    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:47:13.315019 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.314995    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token" pod "1999725a-e166-4784-b3c0-60d11a2bc0fc" (UID: "1999725a-e166-4784-b3c0-60d11a2bc0fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:13.315310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.315290    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo" pod "81284fa4-4933-46ee-9295-767af2657392" (UID: "81284fa4-4933-46ee-9295-767af2657392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:47:13.413342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413273    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81284fa4-4933-46ee-9295-767af2657392-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413310    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413327    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413342    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413356    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413369    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413382    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1999725a-e166-4784-b3c0-60d11a2bc0fc-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413396    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413410    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413424    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413437    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1999725a-e166-4784-b3c0-60d11a2bc0fc-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413451    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qr7\" (UniqueName: \"kubernetes.io/projected/81284fa4-4933-46ee-9295-767af2657392-kube-api-access-j7qr7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413465    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413479    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgrdd\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-kube-api-access-cgrdd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413492    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413505    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81284fa4-4933-46ee-9295-767af2657392-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413518    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81284fa4-4933-46ee-9295-767af2657392-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.413595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.413531    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1999725a-e166-4784-b3c0-60d11a2bc0fc-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:13.850446 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.850413    5533 generic.go:356] "Generic (PLEG): container finished" podID="81284fa4-4933-46ee-9295-767af2657392" containerID="bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5" exitCode=0
Nov 03 21:47:13.850952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.850469    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"
Nov 03 21:47:13.850952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.850502    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" event={"ID":"81284fa4-4933-46ee-9295-767af2657392","Type":"ContainerDied","Data":"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"}
Nov 03 21:47:13.850952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.850563    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl" event={"ID":"81284fa4-4933-46ee-9295-767af2657392","Type":"ContainerDied","Data":"6afd2730ed8430438667c4a5411594614dc7b6e4bdb098fd04e20443bfd269a7"}
Nov 03 21:47:13.850952 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.850602    5533 scope.go:122] "RemoveContainer" containerID="bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"
Nov 03 21:47:13.853433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.853406    5533 generic.go:356] "Generic (PLEG): container finished" podID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerID="d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e" exitCode=0
Nov 03 21:47:13.853565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.853452    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"
Nov 03 21:47:13.854626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.854594    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerDied","Data":"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"}
Nov 03 21:47:13.854786 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.854668    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm" event={"ID":"1999725a-e166-4784-b3c0-60d11a2bc0fc","Type":"ContainerDied","Data":"793efe79f7842087e480caa1eec8def8d13e1f74a0366fe9edd7bd4beff3307d"}
Nov 03 21:47:13.866231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.866215    5533 scope.go:122] "RemoveContainer" containerID="bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"
Nov 03 21:47:13.866700 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:13.866676    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5\": container with ID starting with bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5 not found: ID does not exist" containerID="bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"
Nov 03 21:47:13.866806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.866709    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5"} err="failed to get container status \"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5\": rpc error: code = NotFound desc = could not find container \"bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5\": container with ID starting with bd18afafed7e5fda4379d014a9bb28af80344d352734b1468b395a7b0cd93aa5 not found: ID does not exist"
Nov 03 21:47:13.866806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.866728    5533 scope.go:122] "RemoveContainer" containerID="d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"
Nov 03 21:47:13.879735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.879718    5533 scope.go:122] "RemoveContainer" containerID="164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"
Nov 03 21:47:13.908195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.908177    5533 scope.go:122] "RemoveContainer" containerID="d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"
Nov 03 21:47:13.908436 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:13.908419    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e\": container with ID starting with d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e not found: ID does not exist" containerID="d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"
Nov 03 21:47:13.908485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.908445    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e"} err="failed to get container status \"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e\": rpc error: code = NotFound desc = could not find container \"d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e\": container with ID starting with d948d1da8515caa6ca7f680adfd3be792f40ca44af9ba21966f8ef415d23285e not found: ID does not exist"
Nov 03 21:47:13.908485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.908461    5533 scope.go:122] "RemoveContainer" containerID="164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"
Nov 03 21:47:13.908748 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:13.908724    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29\": container with ID starting with 164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29 not found: ID does not exist" containerID="164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"
Nov 03 21:47:13.908792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:13.908761    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29"} err="failed to get container status \"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29\": rpc error: code = NotFound desc = could not find container \"164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29\": container with ID starting with 164cbebcd01b4ce1e8bc595b2ec592a0614bd7b406cb36cb4be10a5e99d06c29 not found: ID does not exist"
Nov 03 21:47:14.005438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.005411    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:14.007732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.007707    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:47:14.053583 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.053558    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-262zm"]
Nov 03 21:47:14.116002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.115936    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:47:14.126295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.122479    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7da0f7-openshift-default-7b76b8c46f-nvkbl"]
Nov 03 21:47:14.126295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.123215    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\") pod \"701a6078-2cc7-40fd-815f-ed647d10e8b0\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") "
Nov 03 21:47:14.126295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.123268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\" (UniqueName: \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\") pod \"701a6078-2cc7-40fd-815f-ed647d10e8b0\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") "
Nov 03 21:47:14.126295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.123407    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\") pod \"701a6078-2cc7-40fd-815f-ed647d10e8b0\" (UID: \"701a6078-2cc7-40fd-815f-ed647d10e8b0\") "
Nov 03 21:47:14.130572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.130541    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca" pod "701a6078-2cc7-40fd-815f-ed647d10e8b0" (UID: "701a6078-2cc7-40fd-815f-ed647d10e8b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:47:14.131787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.131752    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca" pod "701a6078-2cc7-40fd-815f-ed647d10e8b0" (UID: "701a6078-2cc7-40fd-815f-ed647d10e8b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:47:14.133037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.133012    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh" pod "701a6078-2cc7-40fd-815f-ed647d10e8b0" (UID: "701a6078-2cc7-40fd-815f-ed647d10e8b0"). InnerVolumeSpecName "kube-api-access-fqjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:47:14.225117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.225086    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:14.225210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.225117    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701a6078-2cc7-40fd-815f-ed647d10e8b0-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:14.225210 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.225131    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjjh\" (UniqueName: \"kubernetes.io/projected/701a6078-2cc7-40fd-815f-ed647d10e8b0-kube-api-access-fqjjh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:47:14.497096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.497064    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" path="/var/lib/kubelet/pods/1999725a-e166-4784-b3c0-60d11a2bc0fc/volumes"
Nov 03 21:47:14.497565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.497551    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81284fa4-4933-46ee-9295-767af2657392" path="/var/lib/kubelet/pods/81284fa4-4933-46ee-9295-767af2657392/volumes"
Nov 03 21:47:14.863241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.863155    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"
Nov 03 21:47:14.863241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.863175    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9" event={"ID":"701a6078-2cc7-40fd-815f-ed647d10e8b0","Type":"ContainerDied","Data":"7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5"}
Nov 03 21:47:14.863241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:14.863229    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be15bbc3186f9c9cf06efc87d4fd023d7dc460dcff78886c7d698f6c94969b5"
Nov 03 21:47:19.130392 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.130339    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:47:19.131915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131884    5533 cpu_manager.go:414] "containerMap: removing container" podUID="701a6078-2cc7-40fd-815f-ed647d10e8b0" containerName="revoke-keys"
Nov 03 21:47:19.131915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131914    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701a6078-2cc7-40fd-815f-ed647d10e8b0" containerName="revoke-keys"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131952    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131960    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131978    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.131986    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132012    5533 cpu_manager.go:414] "containerMap: removing container" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy"
Nov 03 21:47:19.132117 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132020    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy"
Nov 03 21:47:19.132413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132400    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="701a6078-2cc7-40fd-815f-ed647d10e8b0" containerName="revoke-keys"
Nov 03 21:47:19.132465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132419    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132428    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1999725a-e166-4784-b3c0-60d11a2bc0fc" containerName="istio-proxy"
Nov 03 21:47:19.132465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.132448    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="81284fa4-4933-46ee-9295-767af2657392" containerName="istio-proxy"
Nov 03 21:47:19.139818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.139781    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.190109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.189838    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:47:19.231592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.231561    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-267053-dockercfg-t6l9w\""
Nov 03 21:47:19.231592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.231577    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-267053\""
Nov 03 21:47:19.294659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.285269    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:47:19.296878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.296437    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz88\" (UniqueName: \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.296878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.296747    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.306783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.306755    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.340275 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:19.340239    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:47:19.351506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.351475    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:47:19.397620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.397529    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.397620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.397579    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.397875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.397760    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz88\" (UniqueName: \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.397875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.397844    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpm54\" (UniqueName: \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.398338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.398318    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.455590 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.455555    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz88\" (UniqueName: \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\") pod \"payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:19.498936 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.498901    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.499119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.499000    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpm54\" (UniqueName: \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.499616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.499593    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.563069 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.563033    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpm54\" (UniqueName: \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\") pod \"payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.636572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.636546    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:19.754251 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:19.754215    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:20.020488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.020455    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:47:20.020720 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:47:20.020694    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e15500_039d_40f5_a602_b6071eec65a8.slice/crio-61ae81b228f7abccc4113df771eafcca1e719fafd47022c50a147b124974397b WatchSource:0}: Error finding container 61ae81b228f7abccc4113df771eafcca1e719fafd47022c50a147b124974397b: Status 404 returned error can't find the container with id 61ae81b228f7abccc4113df771eafcca1e719fafd47022c50a147b124974397b
Nov 03 21:47:20.140985 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.140956    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:47:20.141896 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:47:20.141871    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6552992b_bd0a_4449_a504_cece808d68e3.slice/crio-f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54 WatchSource:0}: Error finding container f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54: Status 404 returned error can't find the container with id f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54
Nov 03 21:47:20.920721 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.920668    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" event={"ID":"6552992b-bd0a-4449-a504-cece808d68e3","Type":"ContainerStarted","Data":"2448a5905ea679a16e24cecec51febff363485614f1ff0db64fc8515f7c0b353"}
Nov 03 21:47:20.920931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.920735    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" event={"ID":"6552992b-bd0a-4449-a504-cece808d68e3","Type":"ContainerStarted","Data":"f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54"}
Nov 03 21:47:20.920931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.920760    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:20.922310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.922278    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" event={"ID":"13e15500-039d-40f5-a602-b6071eec65a8","Type":"ContainerStarted","Data":"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"}
Nov 03 21:47:20.922443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.922328    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:20.922443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:20.922344    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" event={"ID":"13e15500-039d-40f5-a602-b6071eec65a8","Type":"ContainerStarted","Data":"61ae81b228f7abccc4113df771eafcca1e719fafd47022c50a147b124974397b"}
Nov 03 21:47:21.032389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:21.032340    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" podStartSLOduration=2.032327395 podStartE2EDuration="2.032327395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:21.02908365 +0000 UTC m=+3542.660397176" watchObservedRunningTime="2026-11-03 21:47:21.032327395 +0000 UTC m=+3542.663640871"
Nov 03 21:47:21.143737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:21.143690    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" podStartSLOduration=2.1436726520000002 podStartE2EDuration="2.143672652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:21.139979375 +0000 UTC m=+3542.771292854" watchObservedRunningTime="2026-11-03 21:47:21.143672652 +0000 UTC m=+3542.774986133"
Nov 03 21:47:22.331351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.331315    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:47:22.335770 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.335750    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.432797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432765    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:47:22.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432800    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zd6p2\""
Nov 03 21:47:22.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432834    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:47:22.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432868    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:47:22.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432877    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8812cbbcafb8\""
Nov 03 21:47:22.432966 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.432911    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:47:22.434340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.434315    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.434473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.434371    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvtr\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.434473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.434467    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.434678 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.434626    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.436683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.436657    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:47:22.535684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.535654    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.535818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.535715    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvtr\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.535818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.535746    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.535943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.535848    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.536371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.536349    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.536695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.536655    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.538298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.538279    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.593518 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.593459    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvtr\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\") pod \"payload-processing-external-model-8999dc9db-49xvj\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") " pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:22.650323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:22.650300    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:23.021425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:23.021395    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:47:23.022491 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:47:23.022464    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2bc31c_57cc_46b3_96bb_32241fa8bf98.slice/crio-461c3ee42565ce7e7b5596bb5b594d839928cd5056c53f27424f42c0ae03421c WatchSource:0}: Error finding container 461c3ee42565ce7e7b5596bb5b594d839928cd5056c53f27424f42c0ae03421c: Status 404 returned error can't find the container with id 461c3ee42565ce7e7b5596bb5b594d839928cd5056c53f27424f42c0ae03421c
Nov 03 21:47:23.943748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:23.943708    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" event={"ID":"dd2bc31c-57cc-46b3-96bb-32241fa8bf98","Type":"ContainerStarted","Data":"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"}
Nov 03 21:47:23.944183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:23.943773    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" event={"ID":"dd2bc31c-57cc-46b3-96bb-32241fa8bf98","Type":"ContainerStarted","Data":"461c3ee42565ce7e7b5596bb5b594d839928cd5056c53f27424f42c0ae03421c"}
Nov 03 21:47:23.944183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:23.943802    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:24.056500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:24.056454    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" podStartSLOduration=2.056440293 podStartE2EDuration="2.056440293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:24.053374368 +0000 UTC m=+3545.684687845" watchObservedRunningTime="2026-11-03 21:47:24.056440293 +0000 UTC m=+3545.687753812"
Nov 03 21:47:31.931242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:31.931211    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:47:31.931675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:31.931353    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:47:34.953238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:34.953204    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:47:42.435107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:42.435072    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:47:47.050346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:47.050307    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"]
Nov 03 21:47:47.097009 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:47.096977    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce0a736-smdqm"]
Nov 03 21:47:48.494708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:48.494677    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae267d51-0c42-42e1-872a-b8e13fcc0c64" path="/var/lib/kubelet/pods/ae267d51-0c42-42e1-872a-b8e13fcc0c64/volumes"
Nov 03 21:47:49.260478 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:47:49.260446    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:47:57.047053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:57.047007    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"]
Nov 03 21:47:57.093947 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:57.093907    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49b68457-j4mjh"]
Nov 03 21:47:58.507668 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:47:58.507623    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce8cb23-5f00-4d35-b5c8-777c91f71fdf" path="/var/lib/kubelet/pods/2ce8cb23-5f00-4d35-b5c8-777c91f71fdf/volumes"
Nov 03 21:48:06.563310 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:06.563270    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:48:06.563846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:06.563531    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" containerID="cri-o://17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330" gracePeriod=30
Nov 03 21:48:08.155918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:08.155884    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:48:08.156284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:08.155956    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:48:11.826937 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.826909    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:48:11.867727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867665    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.867859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867743    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.867859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867845    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.867989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867876    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.867989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867905    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.867989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867907    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:11.867989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.867965    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.868207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868024    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.868207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868063    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.868207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868104    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\") pod \"4d352da3-493f-4470-88a7-99fba4032d7c\" (UID: \"4d352da3-493f-4470-88a7-99fba4032d7c\") "
Nov 03 21:48:11.868207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868129    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:11.868441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868212    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:11.868441 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868384    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:11.868569 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868452    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:11.868697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868675    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.868784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868704    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d352da3-493f-4470-88a7-99fba4032d7c-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.868784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868720    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.868784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868735    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.868784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.868749    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.870075 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.870047    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:48:11.870188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.870152    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "kube-api-access-fv6m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:11.870345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.870328    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:11.870561 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.870539    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token" pod "4d352da3-493f-4470-88a7-99fba4032d7c" (UID: "4d352da3-493f-4470-88a7-99fba4032d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:11.969273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.969251    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d352da3-493f-4470-88a7-99fba4032d7c-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.969396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.969275    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6m2\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-kube-api-access-fv6m2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.969396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.969286    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d352da3-493f-4470-88a7-99fba4032d7c-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:11.969396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:11.969295    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d352da3-493f-4470-88a7-99fba4032d7c-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:12.297865 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.297833    5533 generic.go:356] "Generic (PLEG): container finished" podID="4d352da3-493f-4470-88a7-99fba4032d7c" containerID="17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330" exitCode=0
Nov 03 21:48:12.298059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.297874    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerDied","Data":"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"}
Nov 03 21:48:12.298059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.297887    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"
Nov 03 21:48:12.298059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.297933    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz" event={"ID":"4d352da3-493f-4470-88a7-99fba4032d7c","Type":"ContainerDied","Data":"8ad12a195e752a07e95261e284dd8c07091583791ef3bdc26085a0cc671d0102"}
Nov 03 21:48:12.298059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.297961    5533 scope.go:122] "RemoveContainer" containerID="17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"
Nov 03 21:48:12.309203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.309186    5533 scope.go:122] "RemoveContainer" containerID="7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"
Nov 03 21:48:12.318679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.318661    5533 scope.go:122] "RemoveContainer" containerID="17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"
Nov 03 21:48:12.318940 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:12.318921    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330\": container with ID starting with 17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330 not found: ID does not exist" containerID="17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"
Nov 03 21:48:12.318995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.318947    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330"} err="failed to get container status \"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330\": rpc error: code = NotFound desc = could not find container \"17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330\": container with ID starting with 17948a01140c36c236ab14cb4809e78e41c61569df1107b5af411c9a81322330 not found: ID does not exist"
Nov 03 21:48:12.318995 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.318964    5533 scope.go:122] "RemoveContainer" containerID="7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"
Nov 03 21:48:12.319185 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:12.319172    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed\": container with ID starting with 7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed not found: ID does not exist" containerID="7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"
Nov 03 21:48:12.319224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.319188    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed"} err="failed to get container status \"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed\": rpc error: code = NotFound desc = could not find container \"7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed\": container with ID starting with 7f22fde61b76d8e2ec73dd35be11550f8bc825624e3af1d7e65b9b5e806117ed not found: ID does not exist"
Nov 03 21:48:12.454864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.454829    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:48:12.457919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.457894    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a3dc77-openshift-default-548879bc69-mxdmz"]
Nov 03 21:48:12.494580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:12.494554    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" path="/var/lib/kubelet/pods/4d352da3-493f-4470-88a7-99fba4032d7c/volumes"
Nov 03 21:48:14.415102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:14.415029    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:48:19.157953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.157926    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:48:19.171060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.171028    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:48:19.196733 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.196709    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:48:19.209276 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.209251    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:48:19.270923 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:19.270884    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:48:19.415854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.415778    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" podUID="e71f5f3d-1f05-4331-93d7-96c27db86cae" containerName="payload-processing" containerID="cri-o://c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432" gracePeriod=25
Nov 03 21:48:19.691615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.691584    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:48:19.710346 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.710314    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:48:19.757427 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.757392    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\") pod \"e71f5f3d-1f05-4331-93d7-96c27db86cae\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") "
Nov 03 21:48:19.757619 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.757603    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\" (UniqueName: \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\") pod \"e71f5f3d-1f05-4331-93d7-96c27db86cae\" (UID: \"e71f5f3d-1f05-4331-93d7-96c27db86cae\") "
Nov 03 21:48:19.757881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.757855    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume" pod "e71f5f3d-1f05-4331-93d7-96c27db86cae" (UID: "e71f5f3d-1f05-4331-93d7-96c27db86cae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:19.758132 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.758112    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71f5f3d-1f05-4331-93d7-96c27db86cae-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:19.759969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.759936    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9" pod "e71f5f3d-1f05-4331-93d7-96c27db86cae" (UID: "e71f5f3d-1f05-4331-93d7-96c27db86cae"). InnerVolumeSpecName "kube-api-access-6mrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:19.846665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.846601    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:48:19.859138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:19.859100    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mrc9\" (UniqueName: \"kubernetes.io/projected/e71f5f3d-1f05-4331-93d7-96c27db86cae-kube-api-access-6mrc9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:20.356541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.356509    5533 generic.go:356] "Generic (PLEG): container finished" podID="e71f5f3d-1f05-4331-93d7-96c27db86cae" containerID="c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432" exitCode=0
Nov 03 21:48:20.356962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.356586    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"
Nov 03 21:48:20.356962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.356587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" event={"ID":"e71f5f3d-1f05-4331-93d7-96c27db86cae","Type":"ContainerDied","Data":"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"}
Nov 03 21:48:20.356962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.356698    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j" event={"ID":"e71f5f3d-1f05-4331-93d7-96c27db86cae","Type":"ContainerDied","Data":"a453a14b1dcf3202cbe752eb9c236f63a6d55b6555dbee9b271e58164607a1eb"}
Nov 03 21:48:20.356962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.356723    5533 scope.go:122] "RemoveContainer" containerID="c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"
Nov 03 21:48:20.367542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.367487    5533 scope.go:122] "RemoveContainer" containerID="c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"
Nov 03 21:48:20.367871 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:20.367817    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432\": container with ID starting with c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432 not found: ID does not exist" containerID="c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"
Nov 03 21:48:20.367871 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.367853    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432"} err="failed to get container status \"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432\": rpc error: code = NotFound desc = could not find container \"c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432\": container with ID starting with c1346def89a794064236f5632715dd230d8f08cabbf153b82da5541a4b6cc432 not found: ID does not exist"
Nov 03 21:48:20.511968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.511940    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:48:20.517814 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:20.517785    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a02404-75dd7cdbfb-cgd5j"]
Nov 03 21:48:21.206335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:21.206304    5533 scope.go:122] "RemoveContainer" containerID="76ba667b6ebee30e237b60f07dd3b16ad52e5086937b4f65c4cca9da397629e8"
Nov 03 21:48:21.217181 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:21.217062    5533 scope.go:122] "RemoveContainer" containerID="26f0074f5818e9fa9561142300096a3468a83a2cb25336480b6bade9b077f2e5"
Nov 03 21:48:22.494687 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:22.494658    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71f5f3d-1f05-4331-93d7-96c27db86cae" path="/var/lib/kubelet/pods/e71f5f3d-1f05-4331-93d7-96c27db86cae/volumes"
Nov 03 21:48:24.711598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:24.711554    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" podUID="fd7102a4-c470-4446-8941-bf0eb614d0f2" containerName="payload-processing" containerID="cri-o://3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3" gracePeriod=25
Nov 03 21:48:24.847692 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:24.847661    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" podUID="343ab693-fc33-41fe-8c73-32c72c033860" containerName="payload-pre-processing" containerID="cri-o://238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19" gracePeriod=25
Nov 03 21:48:24.977287 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:24.977260    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:48:25.119874 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.119845    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\") pod \"fd7102a4-c470-4446-8941-bf0eb614d0f2\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") "
Nov 03 21:48:25.120058 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.120043    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\" (UniqueName: \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\") pod \"fd7102a4-c470-4446-8941-bf0eb614d0f2\" (UID: \"fd7102a4-c470-4446-8941-bf0eb614d0f2\") "
Nov 03 21:48:25.120237 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.120213    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume" pod "fd7102a4-c470-4446-8941-bf0eb614d0f2" (UID: "fd7102a4-c470-4446-8941-bf0eb614d0f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:25.120414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.120397    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7102a4-c470-4446-8941-bf0eb614d0f2-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:25.122631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.122610    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp" pod "fd7102a4-c470-4446-8941-bf0eb614d0f2" (UID: "fd7102a4-c470-4446-8941-bf0eb614d0f2"). InnerVolumeSpecName "kube-api-access-nmgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:25.131230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.131208    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:48:25.221145 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.221085    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\" (UniqueName: \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\") pod \"343ab693-fc33-41fe-8c73-32c72c033860\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") "
Nov 03 21:48:25.221269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.221193    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\") pod \"343ab693-fc33-41fe-8c73-32c72c033860\" (UID: \"343ab693-fc33-41fe-8c73-32c72c033860\") "
Nov 03 21:48:25.221434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.221420    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgpp\" (UniqueName: \"kubernetes.io/projected/fd7102a4-c470-4446-8941-bf0eb614d0f2-kube-api-access-nmgpp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:25.221582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.221562    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume" pod "343ab693-fc33-41fe-8c73-32c72c033860" (UID: "343ab693-fc33-41fe-8c73-32c72c033860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:25.223016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.222997    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89" pod "343ab693-fc33-41fe-8c73-32c72c033860" (UID: "343ab693-fc33-41fe-8c73-32c72c033860"). InnerVolumeSpecName "kube-api-access-4rk89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:25.322473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.322446    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343ab693-fc33-41fe-8c73-32c72c033860-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:25.322473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.322472    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rk89\" (UniqueName: \"kubernetes.io/projected/343ab693-fc33-41fe-8c73-32c72c033860-kube-api-access-4rk89\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:25.397077 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.397048    5533 generic.go:356] "Generic (PLEG): container finished" podID="343ab693-fc33-41fe-8c73-32c72c033860" containerID="238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19" exitCode=0
Nov 03 21:48:25.397244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.397110    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"
Nov 03 21:48:25.397244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.397136    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" event={"ID":"343ab693-fc33-41fe-8c73-32c72c033860","Type":"ContainerDied","Data":"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"}
Nov 03 21:48:25.397244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.397213    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj" event={"ID":"343ab693-fc33-41fe-8c73-32c72c033860","Type":"ContainerDied","Data":"48e2fe55ffb432f62f30d396aa9fc42b1bd32d656e912119b622b355576c503d"}
Nov 03 21:48:25.397244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.397242    5533 scope.go:122] "RemoveContainer" containerID="238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"
Nov 03 21:48:25.398702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.398683    5533 generic.go:356] "Generic (PLEG): container finished" podID="fd7102a4-c470-4446-8941-bf0eb614d0f2" containerID="3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3" exitCode=0
Nov 03 21:48:25.398835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.398723    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" event={"ID":"fd7102a4-c470-4446-8941-bf0eb614d0f2","Type":"ContainerDied","Data":"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"}
Nov 03 21:48:25.398835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.398754    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd" event={"ID":"fd7102a4-c470-4446-8941-bf0eb614d0f2","Type":"ContainerDied","Data":"367be500a484cc827ad3477099d49f1845475e5e58264307a8defe100dd118ca"}
Nov 03 21:48:25.398835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.398761    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"
Nov 03 21:48:25.408906 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.408888    5533 scope.go:122] "RemoveContainer" containerID="238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"
Nov 03 21:48:25.409181 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:25.409162    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19\": container with ID starting with 238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19 not found: ID does not exist" containerID="238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"
Nov 03 21:48:25.409257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.409188    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19"} err="failed to get container status \"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19\": rpc error: code = NotFound desc = could not find container \"238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19\": container with ID starting with 238363dd1349bad6032b8244f35cf1710146bb728c101f54b59f907b8d8b5a19 not found: ID does not exist"
Nov 03 21:48:25.409257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.409205    5533 scope.go:122] "RemoveContainer" containerID="3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"
Nov 03 21:48:25.418306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.418288    5533 scope.go:122] "RemoveContainer" containerID="3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"
Nov 03 21:48:25.418534 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:25.418518    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3\": container with ID starting with 3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3 not found: ID does not exist" containerID="3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"
Nov 03 21:48:25.418580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.418542    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3"} err="failed to get container status \"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3\": rpc error: code = NotFound desc = could not find container \"3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3\": container with ID starting with 3ae9fdf94771a71bc3bcaf261da62bc5659b1771e24390c5196313364e6b60f3 not found: ID does not exist"
Nov 03 21:48:25.555860 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.555763    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:48:25.560483 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.560455    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2057ca-57d44479ff-n8cbj"]
Nov 03 21:48:25.661752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.661727    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:48:25.665787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:25.665760    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2057ca-84bf56d6bd-j94sd"]
Nov 03 21:48:26.496256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:26.496220    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343ab693-fc33-41fe-8c73-32c72c033860" path="/var/lib/kubelet/pods/343ab693-fc33-41fe-8c73-32c72c033860/volumes"
Nov 03 21:48:26.496790 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:26.496769    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7102a4-c470-4446-8941-bf0eb614d0f2" path="/var/lib/kubelet/pods/fd7102a4-c470-4446-8941-bf0eb614d0f2/volumes"
Nov 03 21:48:28.497913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:28.497874    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:48:42.230121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:42.230079    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"]
Nov 03 21:48:42.276828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:42.276794    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c0fc5f-zzpb9"]
Nov 03 21:48:42.495090 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:42.495010    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701a6078-2cc7-40fd-815f-ed647d10e8b0" path="/var/lib/kubelet/pods/701a6078-2cc7-40fd-815f-ed647d10e8b0/volumes"
Nov 03 21:48:45.558115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558085    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"]
Nov 03 21:48:45.558671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558652    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.558671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558671    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558689    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e71f5f3d-1f05-4331-93d7-96c27db86cae" containerName="payload-processing"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558696    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71f5f3d-1f05-4331-93d7-96c27db86cae" containerName="payload-processing"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558705    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558710    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558725    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fd7102a4-c470-4446-8941-bf0eb614d0f2" containerName="payload-processing"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558732    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7102a4-c470-4446-8941-bf0eb614d0f2" containerName="payload-processing"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558745    5533 cpu_manager.go:414] "containerMap: removing container" podUID="343ab693-fc33-41fe-8c73-32c72c033860" containerName="payload-pre-processing"
Nov 03 21:48:45.558845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558751    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343ab693-fc33-41fe-8c73-32c72c033860" containerName="payload-pre-processing"
Nov 03 21:48:45.559129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558945    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.559129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558963    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71f5f3d-1f05-4331-93d7-96c27db86cae" containerName="payload-processing"
Nov 03 21:48:45.559129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558972    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d352da3-493f-4470-88a7-99fba4032d7c" containerName="istio-proxy"
Nov 03 21:48:45.559129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558986    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7102a4-c470-4446-8941-bf0eb614d0f2" containerName="payload-processing"
Nov 03 21:48:45.559129 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.558994    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="343ab693-fc33-41fe-8c73-32c72c033860" containerName="payload-pre-processing"
Nov 03 21:48:45.563991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.563974    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.642597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.642572    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7pp\" (UniqueName: \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.642780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.642757    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.642855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.642835    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.660248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.660222    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:48:45.660248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.660226    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:48:45.663370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.663346    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"]
Nov 03 21:48:45.743623 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.743591    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7pp\" (UniqueName: \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.743773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.743704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.743773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.743734    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.744539 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.744512    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.746307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.746280    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.801402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.801378    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7pp\" (UniqueName: \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\") pod \"maas-api-revoke-sub-cae41440-t7xlp\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:45.878572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:45.878489    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:46.299162 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:46.299128    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"]
Nov 03 21:48:46.300256 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:48:46.300218    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c637588_7799_40c2_a0ea_4e0a6f091c3f.slice/crio-3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a WatchSource:0}: Error finding container 3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a: Status 404 returned error can't find the container with id 3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a
Nov 03 21:48:46.551736 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:46.551664    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp" event={"ID":"4c637588-7799-40c2-a0ea-4e0a6f091c3f","Type":"ContainerStarted","Data":"3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a"}
Nov 03 21:48:47.559854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:47.559817    5533 generic.go:356] "Generic (PLEG): container finished" podID="4c637588-7799-40c2-a0ea-4e0a6f091c3f" containerID="70e49f0d4ff1bbe84ee918b378db9f83cb684b00bafd66d6ec25daf6da8d5f29" exitCode=0
Nov 03 21:48:47.560259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:47.559852    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp" event={"ID":"4c637588-7799-40c2-a0ea-4e0a6f091c3f","Type":"ContainerDied","Data":"70e49f0d4ff1bbe84ee918b378db9f83cb684b00bafd66d6ec25daf6da8d5f29"}
Nov 03 21:48:48.703972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.703948    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:48.776762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.776736    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\") pod \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") "
Nov 03 21:48:48.776894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.776808    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\" (UniqueName: \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\") pod \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") "
Nov 03 21:48:48.776945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.776920    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\") pod \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\" (UID: \"4c637588-7799-40c2-a0ea-4e0a6f091c3f\") "
Nov 03 21:48:48.777093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.777068    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca" pod "4c637588-7799-40c2-a0ea-4e0a6f091c3f" (UID: "4c637588-7799-40c2-a0ea-4e0a6f091c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:48.777301 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.777275    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:48.778910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.778881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp" pod "4c637588-7799-40c2-a0ea-4e0a6f091c3f" (UID: "4c637588-7799-40c2-a0ea-4e0a6f091c3f"). InnerVolumeSpecName "kube-api-access-cf7pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:48.779025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.779005    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca" pod "4c637588-7799-40c2-a0ea-4e0a6f091c3f" (UID: "4c637588-7799-40c2-a0ea-4e0a6f091c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:48:48.878659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.878586    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c637588-7799-40c2-a0ea-4e0a6f091c3f-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:48.878659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:48.878612    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7pp\" (UniqueName: \"kubernetes.io/projected/4c637588-7799-40c2-a0ea-4e0a6f091c3f-kube-api-access-cf7pp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:49.261169 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:48:49.261135    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:48:49.420433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.420403    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:48:49.421139 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.421118    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4c637588-7799-40c2-a0ea-4e0a6f091c3f" containerName="revoke-keys"
Nov 03 21:48:49.421211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.421144    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c637588-7799-40c2-a0ea-4e0a6f091c3f" containerName="revoke-keys"
Nov 03 21:48:49.421307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.421297    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c637588-7799-40c2-a0ea-4e0a6f091c3f" containerName="revoke-keys"
Nov 03 21:48:49.424858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.424837    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.475943 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.475898    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7a7661316685\""
Nov 03 21:48:49.480124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.480096    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:48:49.484627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.484580    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.484783 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.484736    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.484854 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.484789    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhl56\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.484914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.484883    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.576035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.575970    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp" event={"ID":"4c637588-7799-40c2-a0ea-4e0a6f091c3f","Type":"ContainerDied","Data":"3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a"}
Nov 03 21:48:49.576035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.576008    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"
Nov 03 21:48:49.576178 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.576012    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a418a18c56188f44669faae521e4fcc005a20107b0d02ecdc7c65923d45669a"
Nov 03 21:48:49.585931 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.585908    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.586061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.585988    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.586137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.586067    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.586137 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.586104    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhl56\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.586710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.586691    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.586951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.586932    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.588387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.588370    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.643817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.643794    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhl56\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\") pod \"payload-processing-external-model-6d9f6b6f54-bg487\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") " pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:49.738280 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:49.738251    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:50.112071 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:50.112038    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:48:50.112486 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:48:50.112458    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5941c0a0_1a52_405c_ad28_f46e3966b34e.slice/crio-25697e0561c8605dbdd050f7625cfa527a7381ed69e99e4717ddd52422be24bb WatchSource:0}: Error finding container 25697e0561c8605dbdd050f7625cfa527a7381ed69e99e4717ddd52422be24bb: Status 404 returned error can't find the container with id 25697e0561c8605dbdd050f7625cfa527a7381ed69e99e4717ddd52422be24bb
Nov 03 21:48:50.584418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:50.584376    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" event={"ID":"5941c0a0-1a52-405c-ad28-f46e3966b34e","Type":"ContainerStarted","Data":"6004066a12133b235aec088406f6a8bad73436acb954d21592a520e0cd2b80b8"}
Nov 03 21:48:50.584597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:50.584440    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" event={"ID":"5941c0a0-1a52-405c-ad28-f46e3966b34e","Type":"ContainerStarted","Data":"25697e0561c8605dbdd050f7625cfa527a7381ed69e99e4717ddd52422be24bb"}
Nov 03 21:48:50.584708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:50.584674    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:50.696299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:50.696255    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" podStartSLOduration=1.6962426590000002 podStartE2EDuration="1.696242659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:50.694176495 +0000 UTC m=+3632.325489965" watchObservedRunningTime="2026-11-03 21:48:50.696242659 +0000 UTC m=+3632.327556135"
Nov 03 21:48:54.535118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:54.535080    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:48:56.090895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:56.090862    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:48:57.691710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.691674    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:48:57.695236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.695216    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.744487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.744446    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7c432-openshift-default-dockercfg-tw9nz\""
Nov 03 21:48:57.775587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775563    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.775745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775606    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.775745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775662    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.775844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775766    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.775889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775870    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.775938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.775914    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5dx\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.776086 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.776071    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.776124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.776110    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.776196 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.776177    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.796039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.795996    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:48:57.877231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877203    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877247    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877270    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877414 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877387    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877458    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877503    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5dx\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877629    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877679    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877736    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877769    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.877922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877846    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.878039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.877982    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.878201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.878171    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.878316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.878265    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.879605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.879582    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.880255 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.880230    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.936676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.935787    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:57.936676 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:57.936308    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5dx\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\") pod \"e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") " pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:58.011709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.010145    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:58.448675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.448648    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:48:58.450354 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:48:58.450326    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd96c5b1_1787_4cc6_b38e_d9eec976a039.slice/crio-1b1a749269ca683ba633676406dfde7cc138852605af289478e21ee7bb03a73b WatchSource:0}: Error finding container 1b1a749269ca683ba633676406dfde7cc138852605af289478e21ee7bb03a73b: Status 404 returned error can't find the container with id 1b1a749269ca683ba633676406dfde7cc138852605af289478e21ee7bb03a73b
Nov 03 21:48:58.453257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.453222    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:48:58.453363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.453285    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:48:58.453363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.453315    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:48:58.643914 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.643874    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerStarted","Data":"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"}
Nov 03 21:48:58.644112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.643940    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerStarted","Data":"1b1a749269ca683ba633676406dfde7cc138852605af289478e21ee7bb03a73b"}
Nov 03 21:48:58.760493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:58.760388    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podStartSLOduration=1.760372842 podStartE2EDuration="1.760372842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:58.757193798 +0000 UTC m=+3640.388507276" watchObservedRunningTime="2026-11-03 21:48:58.760372842 +0000 UTC m=+3640.391686319"
Nov 03 21:48:59.010587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.010497    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:59.016049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.016020    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:59.535453 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.535403    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" podUID="5941c0a0-1a52-405c-ad28-f46e3966b34e" containerName="payload-processing" containerID="cri-o://6004066a12133b235aec088406f6a8bad73436acb954d21592a520e0cd2b80b8" gracePeriod=25
Nov 03 21:48:59.654390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.654361    5533 generic.go:356] "Generic (PLEG): container finished" podID="5941c0a0-1a52-405c-ad28-f46e3966b34e" containerID="6004066a12133b235aec088406f6a8bad73436acb954d21592a520e0cd2b80b8" exitCode=0
Nov 03 21:48:59.654501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.654437    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" event={"ID":"5941c0a0-1a52-405c-ad28-f46e3966b34e","Type":"ContainerDied","Data":"6004066a12133b235aec088406f6a8bad73436acb954d21592a520e0cd2b80b8"}
Nov 03 21:48:59.654993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.654915    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:59.655831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.655810    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:48:59.809616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.809555    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:48:59.902262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902232    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\") pod \"5941c0a0-1a52-405c-ad28-f46e3966b34e\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") "
Nov 03 21:48:59.902426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902273    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\") pod \"5941c0a0-1a52-405c-ad28-f46e3966b34e\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") "
Nov 03 21:48:59.902426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902347    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\") pod \"5941c0a0-1a52-405c-ad28-f46e3966b34e\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") "
Nov 03 21:48:59.902506 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902470    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\") pod \"5941c0a0-1a52-405c-ad28-f46e3966b34e\" (UID: \"5941c0a0-1a52-405c-ad28-f46e3966b34e\") "
Nov 03 21:48:59.902816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902786    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume" pod "5941c0a0-1a52-405c-ad28-f46e3966b34e" (UID: "5941c0a0-1a52-405c-ad28-f46e3966b34e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:59.902974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.902817    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume" pod "5941c0a0-1a52-405c-ad28-f46e3966b34e" (UID: "5941c0a0-1a52-405c-ad28-f46e3966b34e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:59.903078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.903029    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:59.903078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.903052    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5941c0a0-1a52-405c-ad28-f46e3966b34e-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:48:59.904398 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.904315    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56" pod "5941c0a0-1a52-405c-ad28-f46e3966b34e" (UID: "5941c0a0-1a52-405c-ad28-f46e3966b34e"). InnerVolumeSpecName "kube-api-access-vhl56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:59.904613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:48:59.904594    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials" pod "5941c0a0-1a52-405c-ad28-f46e3966b34e" (UID: "5941c0a0-1a52-405c-ad28-f46e3966b34e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:00.004176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.004144    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhl56\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-kube-api-access-vhl56\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:00.004176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.004173    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5941c0a0-1a52-405c-ad28-f46e3966b34e-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:00.662470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.662439    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"
Nov 03 21:49:00.662470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.662463    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487" event={"ID":"5941c0a0-1a52-405c-ad28-f46e3966b34e","Type":"ContainerDied","Data":"25697e0561c8605dbdd050f7625cfa527a7381ed69e99e4717ddd52422be24bb"}
Nov 03 21:49:00.662809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.662525    5533 scope.go:122] "RemoveContainer" containerID="6004066a12133b235aec088406f6a8bad73436acb954d21592a520e0cd2b80b8"
Nov 03 21:49:00.819683 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.819611    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:49:00.824273 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:00.824250    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d9f6b6f54-bg487"]
Nov 03 21:49:01.091983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.091875    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" podUID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" containerName="payload-processing" containerID="cri-o://7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b" gracePeriod=25
Nov 03 21:49:01.335369 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.335334    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:01.335990 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.335973    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5941c0a0-1a52-405c-ad28-f46e3966b34e" containerName="payload-processing"
Nov 03 21:49:01.336049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.335992    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5941c0a0-1a52-405c-ad28-f46e3966b34e" containerName="payload-processing"
Nov 03 21:49:01.336118 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.336107    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5941c0a0-1a52-405c-ad28-f46e3966b34e" containerName="payload-processing"
Nov 03 21:49:01.339543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.339524    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.378558 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.378536    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:49:01.391792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.391766    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-42d3260f64c3\""
Nov 03 21:49:01.391901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.391888    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-j4wbs\""
Nov 03 21:49:01.398884 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.398855    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:01.418591 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.418569    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2x8\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.418728 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.418606    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.418795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.418730    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.418795 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.418786    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.519425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519395    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\") pod \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") "
Nov 03 21:49:01.519544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519455    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\") pod \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") "
Nov 03 21:49:01.519544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519498    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\") pod \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") "
Nov 03 21:49:01.519544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519533    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\") pod \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\" (UID: \"dd2bc31c-57cc-46b3-96bb-32241fa8bf98\") "
Nov 03 21:49:01.519748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519705    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2x8\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.519819 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519758    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.519878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519851    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.519878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519855    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume" pod "dd2bc31c-57cc-46b3-96bb-32241fa8bf98" (UID: "dd2bc31c-57cc-46b3-96bb-32241fa8bf98"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:01.519878 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519865    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume" pod "dd2bc31c-57cc-46b3-96bb-32241fa8bf98" (UID: "dd2bc31c-57cc-46b3-96bb-32241fa8bf98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:01.520039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.519964    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.520271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.520249    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:01.520335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.520273    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:01.520784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.520760    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.520832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.520765    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.522214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.522190    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr" pod "dd2bc31c-57cc-46b3-96bb-32241fa8bf98" (UID: "dd2bc31c-57cc-46b3-96bb-32241fa8bf98"). InnerVolumeSpecName "kube-api-access-pqvtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:01.522309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.522199    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials" pod "dd2bc31c-57cc-46b3-96bb-32241fa8bf98" (UID: "dd2bc31c-57cc-46b3-96bb-32241fa8bf98"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:01.522309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.522255    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.577428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.577396    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2x8\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\") pod \"payload-processing-external-model-98f6c79c6-7cxd9\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.621035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.620965    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvtr\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-kube-api-access-pqvtr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:01.621035 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.620993    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd2bc31c-57cc-46b3-96bb-32241fa8bf98-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:01.654449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.654419    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:01.673697 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.673620    5533 generic.go:356] "Generic (PLEG): container finished" podID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" containerID="7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b" exitCode=0
Nov 03 21:49:01.673815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.673690    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" event={"ID":"dd2bc31c-57cc-46b3-96bb-32241fa8bf98","Type":"ContainerDied","Data":"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"}
Nov 03 21:49:01.673815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.673769    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj" event={"ID":"dd2bc31c-57cc-46b3-96bb-32241fa8bf98","Type":"ContainerDied","Data":"461c3ee42565ce7e7b5596bb5b594d839928cd5056c53f27424f42c0ae03421c"}
Nov 03 21:49:01.673815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.673800    5533 scope.go:122] "RemoveContainer" containerID="7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"
Nov 03 21:49:01.673970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.673948    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"
Nov 03 21:49:01.685964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.685924    5533 scope.go:122] "RemoveContainer" containerID="7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"
Nov 03 21:49:01.686237 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:01.686213    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b\": container with ID starting with 7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b not found: ID does not exist" containerID="7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"
Nov 03 21:49:01.686315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.686247    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b"} err="failed to get container status \"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b\": rpc error: code = NotFound desc = could not find container \"7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b\": container with ID starting with 7d05f7ecacff22962066175db2008e3054cdaaec461897604603640c6b86b37b not found: ID does not exist"
Nov 03 21:49:01.831304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.831261    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:49:01.834195 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:01.834172    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8999dc9db-49xvj"]
Nov 03 21:49:02.032236 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.032205    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:02.033508 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:02.033477    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc295a9_904d_4f2d_899e_9a7035c60049.slice/crio-26ebea8c1986b0b1b0cf6907fd859ac145f74ee03bcbeb5089c6055a7700219b WatchSource:0}: Error finding container 26ebea8c1986b0b1b0cf6907fd859ac145f74ee03bcbeb5089c6055a7700219b: Status 404 returned error can't find the container with id 26ebea8c1986b0b1b0cf6907fd859ac145f74ee03bcbeb5089c6055a7700219b
Nov 03 21:49:02.495594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.495554    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5941c0a0-1a52-405c-ad28-f46e3966b34e" path="/var/lib/kubelet/pods/5941c0a0-1a52-405c-ad28-f46e3966b34e/volumes"
Nov 03 21:49:02.496052 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.496034    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" path="/var/lib/kubelet/pods/dd2bc31c-57cc-46b3-96bb-32241fa8bf98/volumes"
Nov 03 21:49:02.686593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.686552    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" event={"ID":"1dc295a9-904d-4f2d-899e-9a7035c60049","Type":"ContainerStarted","Data":"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"}
Nov 03 21:49:02.686821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.686608    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" event={"ID":"1dc295a9-904d-4f2d-899e-9a7035c60049","Type":"ContainerStarted","Data":"26ebea8c1986b0b1b0cf6907fd859ac145f74ee03bcbeb5089c6055a7700219b"}
Nov 03 21:49:02.686821 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.686650    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:02.800224 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:02.800123    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" podStartSLOduration=1.8001056050000002 podStartE2EDuration="1.800105605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:02.796283362 +0000 UTC m=+3644.427596840" watchObservedRunningTime="2026-11-03 21:49:02.800105605 +0000 UTC m=+3644.431419083"
Nov 03 21:49:03.844951 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:03.844909    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-42d3260f64c3: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:03.844951 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:03.844942    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:03.845397 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:03.845006    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials podName:1dc295a9-904d-4f2d-899e-9a7035c60049 nodeName:}" failed. No retries permitted until 2026-11-03 21:49:04.344986206 +0000 UTC m=+3645.976299678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials") pod "payload-processing-external-model-98f6c79c6-7cxd9" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049") : secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:04.350831 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:04.350800    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-42d3260f64c3: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:04.350831 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:04.350824    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:04.351065 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:04.350884    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials podName:1dc295a9-904d-4f2d-899e-9a7035c60049 nodeName:}" failed. No retries permitted until 2026-11-03 21:49:05.350869908 +0000 UTC m=+3646.982183364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials") pod "payload-processing-external-model-98f6c79c6-7cxd9" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049") : secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:05.360919 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:05.360886    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-42d3260f64c3: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:05.360919 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:05.360912    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:05.361348 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:05.360968    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials podName:1dc295a9-904d-4f2d-899e-9a7035c60049 nodeName:}" failed. No retries permitted until 2026-11-03 21:49:07.3609526 +0000 UTC m=+3648.992266056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials") pod "payload-processing-external-model-98f6c79c6-7cxd9" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049") : secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:05.994344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:05.994308    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:49:05.995001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:05.994969    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" containerName="payload-processing"
Nov 03 21:49:05.995001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:05.994993    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" containerName="payload-processing"
Nov 03 21:49:05.995241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:05.995141    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2bc31c-57cc-46b3-96bb-32241fa8bf98" containerName="payload-processing"
Nov 03 21:49:05.999827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:05.999803    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.076900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.076869    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.076900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.076911    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9gv\" (UniqueName: \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.099672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.095788    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7c432\""
Nov 03 21:49:06.099672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.096076    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e7c432-dockercfg-6vflc\""
Nov 03 21:49:06.100837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.100812    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:49:06.150831 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.150791    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:49:06.180661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.179707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.180661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.179761    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9gv\" (UniqueName: \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.182597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.180960    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.183011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.182993    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.203958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.202765    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:49:06.237731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.237553    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9gv\" (UniqueName: \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\") pod \"payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.280775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.280712    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.280775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.280759    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.313872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.313843    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:06.382239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.382198    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.382599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.382287    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.383471 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.383424    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.441182 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.441141    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\") pod \"payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.505042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.505008    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:06.704754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.704721    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:49:06.707125 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:06.707091    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0721d28b_4d5c_4772_9434_dba80a2cd05c.slice/crio-5fcf439ae725f19bf36cb20b802636272e2d9d72475124135775a1dad35e9fe3 WatchSource:0}: Error finding container 5fcf439ae725f19bf36cb20b802636272e2d9d72475124135775a1dad35e9fe3: Status 404 returned error can't find the container with id 5fcf439ae725f19bf36cb20b802636272e2d9d72475124135775a1dad35e9fe3
Nov 03 21:49:06.729066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.729044    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" event={"ID":"0721d28b-4d5c-4772-9434-dba80a2cd05c","Type":"ContainerStarted","Data":"5fcf439ae725f19bf36cb20b802636272e2d9d72475124135775a1dad35e9fe3"}
Nov 03 21:49:06.821306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.821237    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:06.903627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:06.903601    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:49:06.904550 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:06.904518    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e68e560_0363_4b83_9b6f_702cf5644ffc.slice/crio-c7d6b5728e253e3e73449d541cdd3f7a8c13dae4d633a40044118e9cc0a6441d WatchSource:0}: Error finding container c7d6b5728e253e3e73449d541cdd3f7a8c13dae4d633a40044118e9cc0a6441d: Status 404 returned error can't find the container with id c7d6b5728e253e3e73449d541cdd3f7a8c13dae4d633a40044118e9cc0a6441d
Nov 03 21:49:07.395055 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:07.395011    5533 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-42d3260f64c3: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:07.395055 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:07.395046    5533 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9: secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:07.395494 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:07.395101    5533 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials podName:1dc295a9-904d-4f2d-899e-9a7035c60049 nodeName:}" failed. No retries permitted until 2026-11-03 21:49:11.395084887 +0000 UTC m=+3653.026398356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials") pod "payload-processing-external-model-98f6c79c6-7cxd9" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049") : secret "agc-cred-42d3260f64c3" not found
Nov 03 21:49:07.748875 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.748821    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" event={"ID":"0721d28b-4d5c-4772-9434-dba80a2cd05c","Type":"ContainerStarted","Data":"fcc74106e9a4e428c1d9e509faafd4eb5f268b504d1faa17d7d3c6ac10c32555"}
Nov 03 21:49:07.750391 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.750362    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:07.753363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.753338    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" event={"ID":"1e68e560-0363-4b83-9b6f-702cf5644ffc","Type":"ContainerStarted","Data":"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"}
Nov 03 21:49:07.753495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.753378    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" event={"ID":"1e68e560-0363-4b83-9b6f-702cf5644ffc","Type":"ContainerStarted","Data":"c7d6b5728e253e3e73449d541cdd3f7a8c13dae4d633a40044118e9cc0a6441d"}
Nov 03 21:49:07.753495 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.753397    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:07.860272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.860160    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" podStartSLOduration=2.860145327 podStartE2EDuration="2.860145327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:07.859152077 +0000 UTC m=+3649.490465555" watchObservedRunningTime="2026-11-03 21:49:07.860145327 +0000 UTC m=+3649.491458804"
Nov 03 21:49:07.971288 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:07.971241    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" podStartSLOduration=1.971228638 podStartE2EDuration="1.971228638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:07.967928354 +0000 UTC m=+3649.599241831" watchObservedRunningTime="2026-11-03 21:49:07.971228638 +0000 UTC m=+3649.602542132"
Nov 03 21:49:11.822096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:11.822041    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" podUID="1dc295a9-904d-4f2d-899e-9a7035c60049" containerName="payload-processing" containerID="cri-o://a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe" gracePeriod=25
Nov 03 21:49:12.195727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.195394    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296230    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\") pod \"1dc295a9-904d-4f2d-899e-9a7035c60049\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") "
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296297    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\") pod \"1dc295a9-904d-4f2d-899e-9a7035c60049\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") "
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296352    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\") pod \"1dc295a9-904d-4f2d-899e-9a7035c60049\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") "
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296482    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\") pod \"1dc295a9-904d-4f2d-899e-9a7035c60049\" (UID: \"1dc295a9-904d-4f2d-899e-9a7035c60049\") "
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296517    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume" pod "1dc295a9-904d-4f2d-899e-9a7035c60049" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.296875    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:12.297663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.297082    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume" pod "1dc295a9-904d-4f2d-899e-9a7035c60049" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:12.298797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.298773    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8" pod "1dc295a9-904d-4f2d-899e-9a7035c60049" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049"). InnerVolumeSpecName "kube-api-access-tl2x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:12.298886 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.298820    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials" pod "1dc295a9-904d-4f2d-899e-9a7035c60049" (UID: "1dc295a9-904d-4f2d-899e-9a7035c60049"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:12.398023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.397936    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2x8\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-kube-api-access-tl2x8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:12.398023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.397967    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dc295a9-904d-4f2d-899e-9a7035c60049-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:12.398023 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.397977    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc295a9-904d-4f2d-899e-9a7035c60049-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:12.803909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.802830    5533 generic.go:356] "Generic (PLEG): container finished" podID="1dc295a9-904d-4f2d-899e-9a7035c60049" containerID="a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe" exitCode=0
Nov 03 21:49:12.803909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.802945    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"
Nov 03 21:49:12.803909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.802956    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" event={"ID":"1dc295a9-904d-4f2d-899e-9a7035c60049","Type":"ContainerDied","Data":"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"}
Nov 03 21:49:12.803909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.803004    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9" event={"ID":"1dc295a9-904d-4f2d-899e-9a7035c60049","Type":"ContainerDied","Data":"26ebea8c1986b0b1b0cf6907fd859ac145f74ee03bcbeb5089c6055a7700219b"}
Nov 03 21:49:12.803909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.803043    5533 scope.go:122] "RemoveContainer" containerID="a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"
Nov 03 21:49:12.818421 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.818398    5533 scope.go:122] "RemoveContainer" containerID="a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"
Nov 03 21:49:12.818797 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:12.818769    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe\": container with ID starting with a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe not found: ID does not exist" containerID="a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"
Nov 03 21:49:12.818910 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.818805    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe"} err="failed to get container status \"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe\": rpc error: code = NotFound desc = could not find container \"a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe\": container with ID starting with a496d196b1256fd8e1781764fe27aa9ec1de8fe15b8d6f427dfb52e3e546a1fe not found: ID does not exist"
Nov 03 21:49:12.981661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:12.980686    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:13.024688 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:13.024173    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-7cxd9"]
Nov 03 21:49:14.495753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.494775    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc295a9-904d-4f2d-899e-9a7035c60049" path="/var/lib/kubelet/pods/1dc295a9-904d-4f2d-899e-9a7035c60049/volumes"
Nov 03 21:49:14.513470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.513442    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:14.514045 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.514029    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1dc295a9-904d-4f2d-899e-9a7035c60049" containerName="payload-processing"
Nov 03 21:49:14.514099 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.514047    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc295a9-904d-4f2d-899e-9a7035c60049" containerName="payload-processing"
Nov 03 21:49:14.514257 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.514243    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc295a9-904d-4f2d-899e-9a7035c60049" containerName="payload-processing"
Nov 03 21:49:14.519194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.519175    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.532477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.532457    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.532594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.532491    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.532594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.532532    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqts7\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.532594 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.532578    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.621231 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621202    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-42d3260f64c3\""
Nov 03 21:49:14.621459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621251    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:49:14.621459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621342    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:49:14.621459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621401    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:49:14.621737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621500    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:49:14.621737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.621434    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l5qcv\""
Nov 03 21:49:14.626307 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.626282    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:14.634015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.633982    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.634112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.634046    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.634190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.634117    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqts7\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.634248 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.634198    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.634996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.634963    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.635114 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.635060    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.638896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.638871    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.707836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.707804    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqts7\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\") pod \"payload-processing-external-model-98f6c79c6-5b56g\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") " pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:14.832333 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:14.832242    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:15.232350 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:15.232317    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:15.233412 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:15.233387    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f710ae_43e4_4460_bbd0_e6c8fcfc16c1.slice/crio-0c34609d3cf74fc55424908c0b50c3e80c5f450133414ca5a8e1228b23194aed WatchSource:0}: Error finding container 0c34609d3cf74fc55424908c0b50c3e80c5f450133414ca5a8e1228b23194aed: Status 404 returned error can't find the container with id 0c34609d3cf74fc55424908c0b50c3e80c5f450133414ca5a8e1228b23194aed
Nov 03 21:49:15.830476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:15.830436    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" event={"ID":"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1","Type":"ContainerStarted","Data":"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"}
Nov 03 21:49:15.830893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:15.830505    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:15.830893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:15.830533    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" event={"ID":"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1","Type":"ContainerStarted","Data":"0c34609d3cf74fc55424908c0b50c3e80c5f450133414ca5a8e1228b23194aed"}
Nov 03 21:49:15.944033 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:15.943977    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" podStartSLOduration=1.943964606 podStartE2EDuration="1.943964606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:15.940590879 +0000 UTC m=+3657.571904355" watchObservedRunningTime="2026-11-03 21:49:15.943964606 +0000 UTC m=+3657.575278082"
Nov 03 21:49:17.420454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:17.420414    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:18.764650 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:18.764605    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:49:19.261865 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:19.261823    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:49:19.328720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.328691    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:49:19.332904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.332887    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.415797 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.415764    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-13c6b6-openshift-default-dockercfg-t5flt\""
Nov 03 21:49:19.422159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.422125    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:49:19.423007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.422978    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423281    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qzv\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423464    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423612 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423598    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423744    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.423902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423876    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.424021 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.423996    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.424211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.424055    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525403    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525444    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525462    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525502    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525529    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525579    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525749    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525780    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.525829 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525813    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qzv\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.526155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525901    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.526155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525951    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.526155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.525981    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.526304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.526214    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.526396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.526373    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.528387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.528367    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.528509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.528458    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.583560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.583528    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qzv\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.583690 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.583570    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\") pod \"e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") " pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.647735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.647694    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:19.780385 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:19.779510    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:49:20.089034 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:20.089002    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2aed43_c8de_4517_9b6b_051f83195ef2.slice/crio-62b2c03925bd5740bab8abc7b44843f87ba8f5122c94fcaae4af186aab27b268 WatchSource:0}: Error finding container 62b2c03925bd5740bab8abc7b44843f87ba8f5122c94fcaae4af186aab27b268: Status 404 returned error can't find the container with id 62b2c03925bd5740bab8abc7b44843f87ba8f5122c94fcaae4af186aab27b268
Nov 03 21:49:20.089322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.089268    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:49:20.092218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.092181    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:20.092317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.092264    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:20.092317 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.092309    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:20.882054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.881968    5533 generic.go:356] "Generic (PLEG): container finished" podID="87039730-f061-4333-a820-7ba0f7de70da" containerID="b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd" exitCode=0
Nov 03 21:49:20.882054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.882035    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerDied","Data":"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"}
Nov 03 21:49:20.882520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.882437    5533 scope.go:122] "RemoveContainer" containerID="b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"
Nov 03 21:49:20.883980 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.883954    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerStarted","Data":"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"}
Nov 03 21:49:20.884134 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.883999    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerStarted","Data":"62b2c03925bd5740bab8abc7b44843f87ba8f5122c94fcaae4af186aab27b268"}
Nov 03 21:49:20.884207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.884154    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:20.884269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.884215    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:20.884269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:20.884252    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:21.113322 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.110908    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podStartSLOduration=2.110888255 podStartE2EDuration="2.110888255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:21.108438101 +0000 UTC m=+3662.739751579" watchObservedRunningTime="2026-11-03 21:49:21.110888255 +0000 UTC m=+3662.742201733"
Nov 03 21:49:21.317300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.317266    5533 scope.go:122] "RemoveContainer" containerID="f67edd6e3f73bef27a880e7a80111f1fc5e3d574ae034e3ee7c34ad2676f24e7"
Nov 03 21:49:21.328578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.328554    5533 scope.go:122] "RemoveContainer" containerID="71a8ca671d926d000740526fbd7bf4ec1ff195a7ed82c07ab447c363054edfe9"
Nov 03 21:49:21.648245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.648171    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:21.673660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.661577    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:21.894475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.893893    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerStarted","Data":"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"}
Nov 03 21:49:21.900883 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.899933    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:21.901229 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:21.901203    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:49:22.828921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:22.828894    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:49:22.829105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:22.829013    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 03 21:49:22.829105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:22.829061    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 03 21:49:22.863675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:22.862689    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" podUID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" containerName="payload-processing" containerID="cri-o://b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1" gracePeriod=25
Nov 03 21:49:23.197766 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.197591    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:23.304681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.304643    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\") pod \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") "
Nov 03 21:49:23.304879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.304841    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\") pod \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") "
Nov 03 21:49:23.304879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.304865    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\") pod \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") "
Nov 03 21:49:23.305020 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.304904    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\") pod \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\" (UID: \"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1\") "
Nov 03 21:49:23.305214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.305186    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume" pod "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" (UID: "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:23.305298 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.305279    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume" pod "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" (UID: "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:23.306880 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.306856    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials" pod "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" (UID: "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:23.306997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.306951    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7" pod "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" (UID: "e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1"). InnerVolumeSpecName "kube-api-access-lqts7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:23.406826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.406753    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:23.406826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.406783    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqts7\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-kube-api-access-lqts7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:23.406826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.406793    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:23.406826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.406803    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:23.829812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.829781    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 03 21:49:23.829986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.829830    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 03 21:49:23.909186 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.909155    5533 generic.go:356] "Generic (PLEG): container finished" podID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" containerID="b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1" exitCode=0
Nov 03 21:49:23.909382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.909200    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"
Nov 03 21:49:23.909382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.909245    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" event={"ID":"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1","Type":"ContainerDied","Data":"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"}
Nov 03 21:49:23.909382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.909306    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g" event={"ID":"e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1","Type":"ContainerDied","Data":"0c34609d3cf74fc55424908c0b50c3e80c5f450133414ca5a8e1228b23194aed"}
Nov 03 21:49:23.909382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.909334    5533 scope.go:122] "RemoveContainer" containerID="b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"
Nov 03 21:49:23.920321 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.920306    5533 scope.go:122] "RemoveContainer" containerID="b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"
Nov 03 21:49:23.920566 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:23.920549    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1\": container with ID starting with b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1 not found: ID does not exist" containerID="b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"
Nov 03 21:49:23.920620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:23.920573    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1"} err="failed to get container status \"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1\": rpc error: code = NotFound desc = could not find container \"b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1\": container with ID starting with b855f06e3f95d1198c3751516277f2a7cb877ac063fecd976fcc59f2036354b1 not found: ID does not exist"
Nov 03 21:49:24.102057 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:24.101975    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:24.148538 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:24.148490    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-98f6c79c6-5b56g"]
Nov 03 21:49:24.495856 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:24.495826    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" path="/var/lib/kubelet/pods/e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1/volumes"
Nov 03 21:49:24.829564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:24.829483    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 03 21:49:24.829564 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:24.829541    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 03 21:49:25.829360 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:25.829321    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 03 21:49:25.829806 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:25.829406    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 03 21:49:26.829747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:26.829709    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 03 21:49:26.830259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:26.829758    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 03 21:49:27.793792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.785497    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:49:27.804698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.804660    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" containerName="payload-processing"
Nov 03 21:49:27.804698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.804694    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" containerName="payload-processing"
Nov 03 21:49:27.804954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.804938    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f710ae-43e4-4460-bbd0-e6c8fcfc16c1" containerName="payload-processing"
Nov 03 21:49:27.811579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.811320    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.845986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.844940    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:49:27.845986 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.845936    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:49:27.858748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.858110    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4j7\" (UniqueName: \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.858748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.858363    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.866976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.866947    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:49:27.905945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.905905    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-13c6b6\""
Nov 03 21:49:27.905945 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.905931    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-13c6b6-dockercfg-gzwm5\""
Nov 03 21:49:27.938730 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.938703    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:49:27.958815 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.958790    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:49:27.959837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.959459    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.959837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.959725    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4j7\" (UniqueName: \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.960499 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.960470    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:27.963338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:27.963316    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.019051 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.019017    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:49:28.020194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.020171    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4j7\" (UniqueName: \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\") pod \"payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") " pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:28.060929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.060867    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjks\" (UniqueName: \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.061081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.061058    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.137107 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.137077    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:28.163565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.162391    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjks\" (UniqueName: \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.163565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.162580    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.163565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.163518    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.224667 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.224581    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjks\" (UniqueName: \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\") pod \"payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.277605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.277571    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.520355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.520323    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:49:28.521993 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:28.521965    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc650e378_4c94_41b2_a05c_e89f4d51ed14.slice/crio-f1e7a54abe72b113f66bb39e87174defdf20111f65f144431d61b39b383b604c WatchSource:0}: Error finding container f1e7a54abe72b113f66bb39e87174defdf20111f65f144431d61b39b383b604c: Status 404 returned error can't find the container with id f1e7a54abe72b113f66bb39e87174defdf20111f65f144431d61b39b383b604c
Nov 03 21:49:28.661264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.661223    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:49:28.674024 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:28.673995    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371d593b_88c7_4bb3_8624_46bed833916b.slice/crio-c9838af0ccb10310d46e36c71030f782b289faa4f30b5cebf81f51ea01e97cea WatchSource:0}: Error finding container c9838af0ccb10310d46e36c71030f782b289faa4f30b5cebf81f51ea01e97cea: Status 404 returned error can't find the container with id c9838af0ccb10310d46e36c71030f782b289faa4f30b5cebf81f51ea01e97cea
Nov 03 21:49:28.946481 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.946444    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" event={"ID":"371d593b-88c7-4bb3-8624-46bed833916b","Type":"ContainerStarted","Data":"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"}
Nov 03 21:49:28.946954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.946507    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" event={"ID":"371d593b-88c7-4bb3-8624-46bed833916b","Type":"ContainerStarted","Data":"c9838af0ccb10310d46e36c71030f782b289faa4f30b5cebf81f51ea01e97cea"}
Nov 03 21:49:28.946954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.946539    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:28.948206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.948179    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" event={"ID":"c650e378-4c94-41b2-a05c-e89f4d51ed14","Type":"ContainerStarted","Data":"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"}
Nov 03 21:49:28.948342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.948225    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" event={"ID":"c650e378-4c94-41b2-a05c-e89f4d51ed14","Type":"ContainerStarted","Data":"f1e7a54abe72b113f66bb39e87174defdf20111f65f144431d61b39b383b604c"}
Nov 03 21:49:28.948342 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:28.948285    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:29.056745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:29.056697    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" podStartSLOduration=2.056682467 podStartE2EDuration="2.056682467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:29.054906774 +0000 UTC m=+3670.686220252" watchObservedRunningTime="2026-11-03 21:49:29.056682467 +0000 UTC m=+3670.687995943"
Nov 03 21:49:29.164438 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:29.164381    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" podStartSLOduration=2.164367131 podStartE2EDuration="2.164367131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:29.163515811 +0000 UTC m=+3670.794829287" watchObservedRunningTime="2026-11-03 21:49:29.164367131 +0000 UTC m=+3670.795680607"
Nov 03 21:49:34.730922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.730888    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 21:49:34.735361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.735345    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839249    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839289    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839402    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2sk\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839444    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839525    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839545    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839563    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839582    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.839938 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.839795    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.928067 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.928041    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 21:49:34.941105 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941074    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941174    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941205    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941295    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2sk\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941358    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941413    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.941993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.941965    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.942112 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.942007    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.942731 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.942703    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.943183 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.943159    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.943349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.943322    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.943504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.943477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.943624 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.943605    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.943921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.943899    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.944921 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.944895    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:34.945693 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:34.945670    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:35.001613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.001557    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2sk\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:35.001613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.001557    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nxpqw\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:35.050002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.049977    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:35.470608 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.470578    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 21:49:35.472408 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:35.472382    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6577519e_ac8f_42bc_81b0_04852356456a.slice/crio-c58c431d4b435fb14b6437fbfcf5b24107a9c4c3fdfa0e2274aa5f7a21dc387d WatchSource:0}: Error finding container c58c431d4b435fb14b6437fbfcf5b24107a9c4c3fdfa0e2274aa5f7a21dc387d: Status 404 returned error can't find the container with id c58c431d4b435fb14b6437fbfcf5b24107a9c4c3fdfa0e2274aa5f7a21dc387d
Nov 03 21:49:35.475074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.475042    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:35.475160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.475107    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:35.475160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.475135    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:35.999897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.999857    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" event={"ID":"6577519e-ac8f-42bc-81b0-04852356456a","Type":"ContainerStarted","Data":"bb80602f0234f9e7a852056e160b600c923e0a4cc04cbc9520e9114a8ebf81c1"}
Nov 03 21:49:36.000389 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:35.999907    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" event={"ID":"6577519e-ac8f-42bc-81b0-04852356456a","Type":"ContainerStarted","Data":"c58c431d4b435fb14b6437fbfcf5b24107a9c4c3fdfa0e2274aa5f7a21dc387d"}
Nov 03 21:49:36.050272 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:36.050230    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:36.051595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:36.051564    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:36.051768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:36.051613    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:36.207532 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:36.207477    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podStartSLOduration=2.207463025 podStartE2EDuration="2.207463025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:36.205662568 +0000 UTC m=+3677.836976050" watchObservedRunningTime="2026-11-03 21:49:36.207463025 +0000 UTC m=+3677.838776502"
Nov 03 21:49:37.066665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:37.062035    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:37.066665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:37.062100    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:37.989012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:37.988972    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:49:37.989271 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:37.989242    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" containerID="cri-o://923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379" gracePeriod=30
Nov 03 21:49:38.050315 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:38.050285    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:38.050466 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:38.050340    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:38.505397 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:38.505354    5533 patch_prober.go:29] interesting pod/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:49:38.505833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:38.505406    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:49:39.068999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.068962    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:39.069939 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.069908    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:39.317967 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.317930    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:49:39.318211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.318183    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy" containerID="cri-o://599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63" gracePeriod=30
Nov 03 21:49:39.982014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.981982    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:49:39.982484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:39.982050    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:49:40.050879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:40.050850    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:40.051140 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:40.051112    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:41.068772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:41.068734    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:41.069370 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:41.069337    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:42.050787 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:42.050746    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:42.051014 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:42.050831    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:43.051409 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.051375    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body=
Nov 03 21:49:43.051869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.051449    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused"
Nov 03 21:49:43.268309 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.268245    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:49:43.396666 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396623    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.396838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396683    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.396838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396711    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.396838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396794    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.396838 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396839    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.397085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396865    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.397085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396896    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:43.397085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.396903    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.397085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397034    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.397295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397168    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:43.397295 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397181    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\") pod \"e16e9124-d79b-4ae9-99e9-1247d686b96b\" (UID: \"e16e9124-d79b-4ae9-99e9-1247d686b96b\") "
Nov 03 21:49:43.397755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397733    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.397926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397910    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.398039 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.397925    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:43.398161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.398108    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:43.398161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.398003    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:43.399451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.399426    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:43.399584 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.399428    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:49:43.399702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.399602    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "kube-api-access-jhnq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:43.399702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.399689    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy" pod "e16e9124-d79b-4ae9-99e9-1247d686b96b" (UID: "e16e9124-d79b-4ae9-99e9-1247d686b96b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:43.499486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499456    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnq7\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-kube-api-access-jhnq7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499482    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499492    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499500    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16e9124-d79b-4ae9-99e9-1247d686b96b-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499509    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499518    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16e9124-d79b-4ae9-99e9-1247d686b96b-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:43.499818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:43.499526    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16e9124-d79b-4ae9-99e9-1247d686b96b-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.070695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.070627    5533 generic.go:356] "Generic (PLEG): container finished" podID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerID="923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379" exitCode=0
Nov 03 21:49:44.070695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.070680    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerDied","Data":"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"}
Nov 03 21:49:44.071299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.070735    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn" event={"ID":"e16e9124-d79b-4ae9-99e9-1247d686b96b","Type":"ContainerDied","Data":"2a399c16a2dc5d86b2068fbf2c19c13e60404230e471d9692e7513d0d3c37432"}
Nov 03 21:49:44.071299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.070757    5533 scope.go:122] "RemoveContainer" containerID="923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"
Nov 03 21:49:44.071299 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.070693    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"
Nov 03 21:49:44.084578 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.084557    5533 scope.go:122] "RemoveContainer" containerID="67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"
Nov 03 21:49:44.097694 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.097626    5533 scope.go:122] "RemoveContainer" containerID="923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"
Nov 03 21:49:44.098122 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:44.098093    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379\": container with ID starting with 923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379 not found: ID does not exist" containerID="923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"
Nov 03 21:49:44.098250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.098129    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379"} err="failed to get container status \"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379\": rpc error: code = NotFound desc = could not find container \"923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379\": container with ID starting with 923bfc2a4216baca5f6eb774f354fa16f3c858ddebefd601386790c55139c379 not found: ID does not exist"
Nov 03 21:49:44.098250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.098154    5533 scope.go:122] "RemoveContainer" containerID="67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"
Nov 03 21:49:44.098444 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:44.098420    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73\": container with ID starting with 67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73 not found: ID does not exist" containerID="67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"
Nov 03 21:49:44.098522 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.098451    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73"} err="failed to get container status \"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73\": rpc error: code = NotFound desc = could not find container \"67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73\": container with ID starting with 67d65f89ddb13f95b77fb97e2415119dc570c6c39d8d5e2298a0294a90cd0c73 not found: ID does not exist"
Nov 03 21:49:44.180798 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.180770    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:44.181331 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.181301    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:44.182426 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.182408    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 21:49:44.228012 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.227987    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:49:44.233049 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.233021    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2057ca-openshift-default-6468f86b9-phtfn"]
Nov 03 21:49:44.497439 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.497408    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" path="/var/lib/kubelet/pods/e16e9124-d79b-4ae9-99e9-1247d686b96b/volumes"
Nov 03 21:49:44.591259 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.591206    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:49:44.718958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.718920    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719010    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719058    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719119    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719165 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719149    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719203    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719281    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:44.719425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719334    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719427    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719605 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719482    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\") pod \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\" (UID: \"b6cf8c53-6af5-41aa-aa39-8aced313cb23\") "
Nov 03 21:49:44.719753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.719608    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:49:44.720091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.720025    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.720091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.720053    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.720290 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.720169    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:44.720767 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.720743    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:44.720951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.720927    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:44.723445 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.723420    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:49:44.723758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.723736    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:49:44.724000 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.723978    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "kube-api-access-2d9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:44.724459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.724438    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token" pod "b6cf8c53-6af5-41aa-aa39-8aced313cb23" (UID: "b6cf8c53-6af5-41aa-aa39-8aced313cb23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:49:44.821563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821528    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9xb\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-kube-api-access-2d9xb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821563 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821561    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821576    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821589    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821603    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821617    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cf8c53-6af5-41aa-aa39-8aced313cb23-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:44.821800 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:44.821630    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cf8c53-6af5-41aa-aa39-8aced313cb23-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:49:45.080566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.080534    5533 generic.go:356] "Generic (PLEG): container finished" podID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerID="599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63" exitCode=0
Nov 03 21:49:45.080983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.080586    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"
Nov 03 21:49:45.080983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.080620    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerDied","Data":"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"}
Nov 03 21:49:45.080983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.080699    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n" event={"ID":"b6cf8c53-6af5-41aa-aa39-8aced313cb23","Type":"ContainerDied","Data":"13ce8bb8900f20fe588bb0584fedebaf4044a3899685cda7159f48456243d87a"}
Nov 03 21:49:45.080983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.080723    5533 scope.go:122] "RemoveContainer" containerID="599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"
Nov 03 21:49:45.091652 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.091621    5533 scope.go:122] "RemoveContainer" containerID="ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"
Nov 03 21:49:45.102201 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.102169    5533 scope.go:122] "RemoveContainer" containerID="599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"
Nov 03 21:49:45.102494 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:45.102470    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63\": container with ID starting with 599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63 not found: ID does not exist" containerID="599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"
Nov 03 21:49:45.102572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.102508    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63"} err="failed to get container status \"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63\": rpc error: code = NotFound desc = could not find container \"599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63\": container with ID starting with 599a753659cb9ba1ecb1f576ed416f093b8e5870305d622f89426c44c9ef3a63 not found: ID does not exist"
Nov 03 21:49:45.102572 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.102525    5533 scope.go:122] "RemoveContainer" containerID="ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"
Nov 03 21:49:45.102846 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:45.102824    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b\": container with ID starting with ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b not found: ID does not exist" containerID="ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"
Nov 03 21:49:45.102909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.102851    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b"} err="failed to get container status \"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b\": rpc error: code = NotFound desc = could not find container \"ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b\": container with ID starting with ef53195ac1194bfba5422782b7e6e2035266186c0038b107ad6645cc0e96223b not found: ID does not exist"
Nov 03 21:49:45.242589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.242566    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:49:45.288993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:45.288964    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02404-openshift-default-555fb7868f-64w9n"]
Nov 03 21:49:46.497179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:46.497142    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" path="/var/lib/kubelet/pods/b6cf8c53-6af5-41aa-aa39-8aced313cb23/volumes"
Nov 03 21:49:49.261555 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:49:49.261507    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:49:49.833926 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.833894    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 21:49:49.834476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834462    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.834545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834476    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.834545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834501    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.834545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834508    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.834729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834613    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.834729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834626    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.834729 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.834647    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.839463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.839445    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.880896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.880867    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 21:49:49.881496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.881473    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.881682 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.881654    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16e9124-d79b-4ae9-99e9-1247d686b96b" containerName="istio-proxy"
Nov 03 21:49:49.881804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.881713    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.881804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.881724    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.881887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.881866    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cf8c53-6af5-41aa-aa39-8aced313cb23" containerName="istio-proxy"
Nov 03 21:49:49.882844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.882820    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.882951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.882857    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.882982    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883025 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883014    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883135 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883055    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883179 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883152    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883230 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883206    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883349 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883332    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.883410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.883370    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58m6\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.885243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.885228    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.936358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.936335    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 21:49:49.996062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.996028    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 21:49:49.996792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.996766    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.996905 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.996818    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.996962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.996898    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58m6\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.997119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997095    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.997540 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997519    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.997753 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997726    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.997872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997578    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.997872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997763    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.997872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997850    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997895    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997921    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997954    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.997982    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998004    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998028    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998072 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998069    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998106    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998115    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998150    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998190    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k442j\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998249    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:49.998434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998374    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:49.999026 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:49.998997    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.001053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.001029    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.001170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.001144    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.054568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.054539    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58m6\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.055337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.055311    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-tp7d5\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.099125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099047    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k442j\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099099    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099142    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099280    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099320    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099414    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099457    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099501    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099542    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.099830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.099793    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.100068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.100041    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.100189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.100111    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.100293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.100252    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.100293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.100277    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.101918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.101896    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.102684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.102664    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.153527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.153262    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:50.161403 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.161351    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.169550 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.169487    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k442j\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-4hs5z\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.198862 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.198620    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:50.589242 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:50.589188    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc6d9ee_99df_4755_95d7_6e5196567e04.slice/crio-f4e6451b0b8f06e19cb33b2d3cc5e160eccc84ff516347d6baab4235837b4413 WatchSource:0}: Error finding container f4e6451b0b8f06e19cb33b2d3cc5e160eccc84ff516347d6baab4235837b4413: Status 404 returned error can't find the container with id f4e6451b0b8f06e19cb33b2d3cc5e160eccc84ff516347d6baab4235837b4413
Nov 03 21:49:50.595128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.594517    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.595128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.594608    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.595128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.594685    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.600718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.600093    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 21:49:50.627109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.627083    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:49:50.633620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.633552    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.635505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.635477    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 21:49:50.635606 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:50.635537    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908a4919_eb96_41ec_8ae7_9c7ea103e288.slice/crio-70a8dcecc7478e1a5dbc971aafa28a170881a031469446f5bae00b37d844a8b9 WatchSource:0}: Error finding container 70a8dcecc7478e1a5dbc971aafa28a170881a031469446f5bae00b37d844a8b9: Status 404 returned error can't find the container with id 70a8dcecc7478e1a5dbc971aafa28a170881a031469446f5bae00b37d844a8b9
Nov 03 21:49:50.638130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.638100    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.638211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.638163    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.638256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.638207    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:49:50.685101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.684357    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-965fe8\""
Nov 03 21:49:50.685101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.685019    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-965fe8-dockercfg-sdxt5\""
Nov 03 21:49:50.688313 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.688084    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:49:50.709448 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.709420    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6ps\" (UniqueName: \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.709789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.709695    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.763663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.763269    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:49:50.787626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.785904    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.818669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.817708    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.818669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.817762    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.818669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.817878    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vj5\" (UniqueName: \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.818669 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.817946    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6ps\" (UniqueName: \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.819580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.819554    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.835691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.830354    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:49:50.885503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.885456    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6ps\" (UniqueName: \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\") pod \"payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.919399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.919359    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vj5\" (UniqueName: \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.919604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.919520    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.920233 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.920205    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:50.972485 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.972447    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:50.981223 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:50.981190    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vj5\" (UniqueName: \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\") pod \"payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") " pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:51.137912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.137757    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" event={"ID":"908a4919-eb96-41ec-8ae7-9c7ea103e288","Type":"ContainerStarted","Data":"38daf7b7e35dd219872aaf4fd10265f82b5a1c873b29579b7aee6369744395db"}
Nov 03 21:49:51.137912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.137820    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" event={"ID":"908a4919-eb96-41ec-8ae7-9c7ea103e288","Type":"ContainerStarted","Data":"70a8dcecc7478e1a5dbc971aafa28a170881a031469446f5bae00b37d844a8b9"}
Nov 03 21:49:51.140188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.139911    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" event={"ID":"2dc6d9ee-99df-4755-95d7-6e5196567e04","Type":"ContainerStarted","Data":"c354991d52b2fddd747aa54a8daee909d8cff1cb11fc157cd5ee74574593260d"}
Nov 03 21:49:51.140188 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.139971    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" event={"ID":"2dc6d9ee-99df-4755-95d7-6e5196567e04","Type":"ContainerStarted","Data":"f4e6451b0b8f06e19cb33b2d3cc5e160eccc84ff516347d6baab4235837b4413"}
Nov 03 21:49:51.154604 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.154569    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:49:51.156015 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.155701    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:51.156142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.156050    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:51.156142 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.156088    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:51.200684 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.199507    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:49:51.200882 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.200855    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:51.200968 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.200906    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:51.364124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.364063    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podStartSLOduration=2.364041763 podStartE2EDuration="2.364041763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:51.260852601 +0000 UTC m=+3692.892166080" watchObservedRunningTime="2026-11-03 21:49:51.364041763 +0000 UTC m=+3692.995355241"
Nov 03 21:49:51.365911 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:51.365849    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfab88c_7b77_4cdd_8870_57a86a702162.slice/crio-71eb214e4ab60d7c9754a50390028dbaa5915144aa6eced4e3c84830d9b2fc1e WatchSource:0}: Error finding container 71eb214e4ab60d7c9754a50390028dbaa5915144aa6eced4e3c84830d9b2fc1e: Status 404 returned error can't find the container with id 71eb214e4ab60d7c9754a50390028dbaa5915144aa6eced4e3c84830d9b2fc1e
Nov 03 21:49:51.366857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.366828    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:49:51.378917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.378871    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podStartSLOduration=2.378858287 podStartE2EDuration="2.378858287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:51.376974563 +0000 UTC m=+3693.008288040" watchObservedRunningTime="2026-11-03 21:49:51.378858287 +0000 UTC m=+3693.010171764"
Nov 03 21:49:51.554994 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:49:51.554967    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e732f3_e9c1_45d2_b1f3_5f4ecfeb6b80.slice/crio-7b9bfbfba6c5d2ef97f30a839906227ad4819964ef80ac7a34c5734802cb020b WatchSource:0}: Error finding container 7b9bfbfba6c5d2ef97f30a839906227ad4819964ef80ac7a34c5734802cb020b: Status 404 returned error can't find the container with id 7b9bfbfba6c5d2ef97f30a839906227ad4819964ef80ac7a34c5734802cb020b
Nov 03 21:49:51.555857 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:51.555832    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:49:52.169662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.155007    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:52.169662 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.155097    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:52.212670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.199759    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:52.212670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.199815    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:52.221235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.220843    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" event={"ID":"2dfab88c-7b77-4cdd-8870-57a86a702162","Type":"ContainerStarted","Data":"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"}
Nov 03 21:49:52.221235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.220906    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" event={"ID":"2dfab88c-7b77-4cdd-8870-57a86a702162","Type":"ContainerStarted","Data":"71eb214e4ab60d7c9754a50390028dbaa5915144aa6eced4e3c84830d9b2fc1e"}
Nov 03 21:49:52.221235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.220956    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:49:52.224088 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.224037    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" event={"ID":"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80","Type":"ContainerStarted","Data":"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"}
Nov 03 21:49:52.224209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.224106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" event={"ID":"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80","Type":"ContainerStarted","Data":"7b9bfbfba6c5d2ef97f30a839906227ad4819964ef80ac7a34c5734802cb020b"}
Nov 03 21:49:52.224826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.224805    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:49:52.340570 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.340512    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" podStartSLOduration=2.340496575 podStartE2EDuration="2.340496575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:52.33720317 +0000 UTC m=+3693.968516647" watchObservedRunningTime="2026-11-03 21:49:52.340496575 +0000 UTC m=+3693.971810099"
Nov 03 21:49:52.449487 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:52.449434    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" podStartSLOduration=2.449421146 podStartE2EDuration="2.449421146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:52.447696646 +0000 UTC m=+3694.079010140" watchObservedRunningTime="2026-11-03 21:49:52.449421146 +0000 UTC m=+3694.080734623"
Nov 03 21:49:53.168659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:53.154964    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:53.168659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:53.155022    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:53.204659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:53.200473    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:53.204659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:53.200530    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:54.161661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:54.155058    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:54.161661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:54.155136    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:54.202658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:54.201985    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:54.202658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:54.202053    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:55.172659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:55.166309    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:55.172659 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:55.166369    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:55.209660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:55.208913    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:55.209660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:55.208969    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:56.158661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:56.154943    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:56.158661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:56.155017    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:56.202660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:56.201923    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:56.202660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:56.201993    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:57.165660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:57.163680    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:57.165660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:57.163750    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:57.200656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:57.200112    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:57.200656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:57.200171    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:58.153818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:58.153775    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 21:49:58.154442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:58.153827    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 21:49:58.199991 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:58.199958    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Nov 03 21:49:58.200241 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:58.200204    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Nov 03 21:49:58.686699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:49:58.686656    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:50:00.163762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.163725    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:50:00.167679 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.167628    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:50:00.210335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.210301    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:50:00.210621 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.210587    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:50:00.267710 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.267674    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:50:00.270163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.270139    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:50:00.280746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:00.280722    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 21:50:01.215751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:01.215713    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:50:01.216364 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:01.216328    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:50:01.225881 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:01.225856    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:50:01.228335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:01.228315    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:50:01.230262 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:01.230241    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 21:50:03.234630 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:03.234598    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:50:03.235184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:03.234704    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:50:13.916664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:13.916556    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:50:13.923002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:13.922973    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:13.977665 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:13.977502    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:50:14.016627 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.016582    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-76eaf8\""
Nov 03 21:50:14.016852 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.016689    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-76eaf8-dockercfg-6zw59\""
Nov 03 21:50:14.040503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.040473    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.040698 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.040521    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sphq\" (UniqueName: \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.141872 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.141840    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.142061 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.141892    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sphq\" (UniqueName: \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.142555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.142533    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.200559 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.200530    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sphq\" (UniqueName: \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\") pod \"payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.248133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.248098    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:14.627928 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:14.627896    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:50:14.628236 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:50:14.628211    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b58f63d_506b_4277_8948_4fff8fc3869e.slice/crio-b7a54ebccf91e0cee81e6cf0031a78c88b04a5eef832acfcd93825790e7e37a0 WatchSource:0}: Error finding container b7a54ebccf91e0cee81e6cf0031a78c88b04a5eef832acfcd93825790e7e37a0: Status 404 returned error can't find the container with id b7a54ebccf91e0cee81e6cf0031a78c88b04a5eef832acfcd93825790e7e37a0
Nov 03 21:50:15.525064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:15.525023    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" event={"ID":"5b58f63d-506b-4277-8948-4fff8fc3869e","Type":"ContainerStarted","Data":"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"}
Nov 03 21:50:15.525459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:15.525086    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" event={"ID":"5b58f63d-506b-4277-8948-4fff8fc3869e","Type":"ContainerStarted","Data":"b7a54ebccf91e0cee81e6cf0031a78c88b04a5eef832acfcd93825790e7e37a0"}
Nov 03 21:50:15.525459 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:15.525192    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:15.636330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:15.636262    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" podStartSLOduration=2.636245708 podStartE2EDuration="2.636245708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:15.63374437 +0000 UTC m=+3717.265057925" watchObservedRunningTime="2026-11-03 21:50:15.636245708 +0000 UTC m=+3717.267559187"
Nov 03 21:50:19.261968 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:50:19.261927    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:50:26.534046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:26.534016    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:50:49.262315 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:50:49.262275    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:50:50.073706 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:50:50.073674    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:51:00.554006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.553972    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": read tcp 10.133.0.2:39960->10.133.1.29:15021: read: connection reset by peer" start-of-body=
Nov 03 21:51:00.554443 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.554043    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": read tcp 10.133.0.2:39960->10.133.1.29:15021: read: connection reset by peer"
Nov 03 21:51:00.941581 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.941552    5533 generic.go:356] "Generic (PLEG): container finished" podID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerID="33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292" exitCode=0
Nov 03 21:51:00.941738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.941617    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerDied","Data":"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"}
Nov 03 21:51:00.941983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.941969    5533 scope.go:122] "RemoveContainer" containerID="33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"
Nov 03 21:51:00.943573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.943543    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:00.943707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.943609    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:00.943707 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:00.943671    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:01.971158 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:01.971106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerStarted","Data":"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"}
Nov 03 21:51:02.010595 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:02.010560    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:51:02.010805 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:02.010784    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 21:51:02.010858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:02.010832    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 21:51:03.010859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:03.010822    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 21:51:03.011347 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:03.010876    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 21:51:04.011394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:04.011354    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 21:51:04.011808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:04.011417    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 21:51:05.011125 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.011090    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 21:51:05.011303 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.011144    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 21:51:05.067658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.058109    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:51:05.089206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.089174    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.140473 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.140444    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e9142-openshift-default-dockercfg-h8hp8\""
Nov 03 21:51:05.157498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.157452    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.157836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.157816    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.158010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.157990    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.247734    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.247793    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.247840    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.247956    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.248016    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.248209 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.248092    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xks9t\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.268792 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.268709    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:51:05.349516 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349474    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xks9t\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349598    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349675    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349789    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349835 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349820    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349909 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349859    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349946 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349925    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.349984 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.349974    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.350363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.350340    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.350587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.350563    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.350701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.350569    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.350773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.350707    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.350773 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.350753    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.383812 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.382972    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.389053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.388955    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.407336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.407312    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xks9t\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.421593 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.421567    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\") pod \"e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") " pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.437598 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.437574    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:05.890400 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.890237    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:51:05.890708 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:51:05.890668    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5c70e4_1df2_46e9_93c8_527e004f7aa8.slice/crio-fe89e4d0f018b8313cc0d23fea35cab6a5ddda160bae62966014ed2422eee27e WatchSource:0}: Error finding container fe89e4d0f018b8313cc0d23fea35cab6a5ddda160bae62966014ed2422eee27e: Status 404 returned error can't find the container with id fe89e4d0f018b8313cc0d23fea35cab6a5ddda160bae62966014ed2422eee27e
Nov 03 21:51:05.893648 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.893602    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:05.893757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.893689    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:05.893757 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:05.893719    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:06.014675 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.014602    5533 patch_prober.go:29] interesting pod/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body=
Nov 03 21:51:06.014859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.014684    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused"
Nov 03 21:51:06.016270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.016244    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerStarted","Data":"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"}
Nov 03 21:51:06.016390 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.016300    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerStarted","Data":"fe89e4d0f018b8313cc0d23fea35cab6a5ddda160bae62966014ed2422eee27e"}
Nov 03 21:51:06.133002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.132949    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podStartSLOduration=1.1329349 podStartE2EDuration="1.1329349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:06.129933196 +0000 UTC m=+3767.761246673" watchObservedRunningTime="2026-11-03 21:51:06.1329349 +0000 UTC m=+3767.764248377"
Nov 03 21:51:06.438484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.438230    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:06.439794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.439765    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:51:06.439957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:06.439812    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:51:07.119410 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:07.119379    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:51:07.119765 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:07.119742    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:51:07.121585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:07.121560    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:51:07.443068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:07.443036    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:08.033848 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:08.033814    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:08.034704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:08.034681    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:51:19.261572 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:51:19.261529    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:51:22.689250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:22.689219    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": read tcp 10.133.0.2:57878->10.133.1.34:15021: read: connection reset by peer" start-of-body=
Nov 03 21:51:22.689746 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:22.689287    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": read tcp 10.133.0.2:57878->10.133.1.34:15021: read: connection reset by peer"
Nov 03 21:51:23.163750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.163673    5533 generic.go:356] "Generic (PLEG): container finished" podID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerID="35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f" exitCode=0
Nov 03 21:51:23.163913 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.163746    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerDied","Data":"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"}
Nov 03 21:51:23.164155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.164136    5533 scope.go:122] "RemoveContainer" containerID="35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"
Nov 03 21:51:23.165816 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.165786    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:23.165962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.165866    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:23.165962 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:23.165912    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:51:24.175461 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:24.175423    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerStarted","Data":"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"}
Nov 03 21:51:24.648747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:24.648629    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:51:24.648747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:24.648736    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:24.648993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:24.648783    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:25.648215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:25.648182    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:25.648589 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:25.648230    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:26.648866 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:26.648834    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:26.649243 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:26.648892    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:27.648755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:27.648713    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:27.648969 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:27.648770    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:28.648889 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:28.648853    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:28.649207 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:28.648908    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:29.648407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:29.648326    5533 patch_prober.go:29] interesting pod/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body=
Nov 03 21:51:29.648407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:29.648373    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused"
Nov 03 21:51:30.655060 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:30.655020    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:51:30.655580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:30.655513    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:51:30.656367 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:30.656342    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:51:32.643292 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:51:32.643257    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:51:49.263474 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:51:49.263432    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:52:05.857465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:05.857428    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:52:05.863047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:05.863018    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:05.928081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:05.928046    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:52:05.936335 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:05.936304    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a0da1\""
Nov 03 21:52:05.972062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:05.972030    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a0da1-dockercfg-4t86q\""
Nov 03 21:52:06.031896 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.031864    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:52:06.036316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.036291    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.036597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.036571    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4xj\" (UniqueName: \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.036793 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.036763    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.091768 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.091738    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:52:06.138270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.138210    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6vb\" (UniqueName: \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.138270 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.138254    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.138432 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.138360    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4xj\" (UniqueName: \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.138475 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.138447    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.139127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.139110    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.196712 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.196675    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4xj\" (UniqueName: \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\") pod \"payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.239933 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.239906    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6vb\" (UniqueName: \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.240089 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.239949    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.240626 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.240605    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.298085 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.298055    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6vb\" (UniqueName: \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\") pod \"payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.350376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.350352    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:06.494955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.494913    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:06.733749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.733723    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:52:06.734898 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:52:06.734871    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f94dfa1_639f_4e4d_8104_64059edec661.slice/crio-42edc467f6790759183540cf4018a1aad271f2593a3a402f019cd3c973dc6a3a WatchSource:0}: Error finding container 42edc467f6790759183540cf4018a1aad271f2593a3a402f019cd3c973dc6a3a: Status 404 returned error can't find the container with id 42edc467f6790759183540cf4018a1aad271f2593a3a402f019cd3c973dc6a3a
Nov 03 21:52:06.874929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:06.874903    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:52:06.876192 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:52:06.876166    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf930e11a_c9d6_40b4_b89f_c266e337addf.slice/crio-ec0b18f52cca8478329214f825c209a48876687d9cd99fac2aa18e4f22854887 WatchSource:0}: Error finding container ec0b18f52cca8478329214f825c209a48876687d9cd99fac2aa18e4f22854887: Status 404 returned error can't find the container with id ec0b18f52cca8478329214f825c209a48876687d9cd99fac2aa18e4f22854887
Nov 03 21:52:07.527993 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.527960    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" event={"ID":"5f94dfa1-639f-4e4d-8104-64059edec661","Type":"ContainerStarted","Data":"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"}
Nov 03 21:52:07.528200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.528012    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" event={"ID":"5f94dfa1-639f-4e4d-8104-64059edec661","Type":"ContainerStarted","Data":"42edc467f6790759183540cf4018a1aad271f2593a3a402f019cd3c973dc6a3a"}
Nov 03 21:52:07.528200 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.528049    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:07.529681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.529654    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" event={"ID":"f930e11a-c9d6-40b4-b89f-c266e337addf","Type":"ContainerStarted","Data":"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"}
Nov 03 21:52:07.529807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.529703    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:07.529807 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.529725    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" event={"ID":"f930e11a-c9d6-40b4-b89f-c266e337addf","Type":"ContainerStarted","Data":"ec0b18f52cca8478329214f825c209a48876687d9cd99fac2aa18e4f22854887"}
Nov 03 21:52:07.638965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.638918    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" podStartSLOduration=2.638904898 podStartE2EDuration="2.638904898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:07.636628656 +0000 UTC m=+3829.267942133" watchObservedRunningTime="2026-11-03 21:52:07.638904898 +0000 UTC m=+3829.270218399"
Nov 03 21:52:07.748864 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:07.748815    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" podStartSLOduration=2.7488018800000003 podStartE2EDuration="2.74880188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:07.745745053 +0000 UTC m=+3829.377058541" watchObservedRunningTime="2026-11-03 21:52:07.74880188 +0000 UTC m=+3829.380115413"
Nov 03 21:52:18.540772 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:18.540740    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:52:18.541284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:18.541265    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:52:19.261233 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:19.261199    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:52:28.152425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:28.152332    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:52:29.688504 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:29.688466    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:52:29.794425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:29.794393    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:52:34.689036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:34.688961    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" podUID="c650e378-4c94-41b2-a05c-e89f4d51ed14" containerName="payload-processing" containerID="cri-o://574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779" gracePeriod=25
Nov 03 21:52:34.795863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:34.795821    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" podUID="371d593b-88c7-4bb3-8624-46bed833916b" containerName="payload-pre-processing" containerID="cri-o://31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b" gracePeriod=25
Nov 03 21:52:34.973323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:34.973296    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:52:35.076101 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.076074    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:52:35.111573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.111544    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\" (UniqueName: \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\") pod \"c650e378-4c94-41b2-a05c-e89f4d51ed14\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") "
Nov 03 21:52:35.111809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.111792    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\") pod \"c650e378-4c94-41b2-a05c-e89f4d51ed14\" (UID: \"c650e378-4c94-41b2-a05c-e89f4d51ed14\") "
Nov 03 21:52:35.112232 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.112204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume" pod "c650e378-4c94-41b2-a05c-e89f4d51ed14" (UID: "c650e378-4c94-41b2-a05c-e89f4d51ed14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:35.112482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.112456    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650e378-4c94-41b2-a05c-e89f4d51ed14-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:35.113842 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.113815    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7" pod "c650e378-4c94-41b2-a05c-e89f4d51ed14" (UID: "c650e378-4c94-41b2-a05c-e89f4d51ed14"). InnerVolumeSpecName "kube-api-access-zz4j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:35.213484 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.213408    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\") pod \"371d593b-88c7-4bb3-8624-46bed833916b\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") "
Nov 03 21:52:35.213691 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.213631    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\" (UniqueName: \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\") pod \"371d593b-88c7-4bb3-8624-46bed833916b\" (UID: \"371d593b-88c7-4bb3-8624-46bed833916b\") "
Nov 03 21:52:35.213887 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.213865    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume" pod "371d593b-88c7-4bb3-8624-46bed833916b" (UID: "371d593b-88c7-4bb3-8624-46bed833916b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:35.214189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.214170    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371d593b-88c7-4bb3-8624-46bed833916b-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:35.214266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.214195    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4j7\" (UniqueName: \"kubernetes.io/projected/c650e378-4c94-41b2-a05c-e89f4d51ed14-kube-api-access-zz4j7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:35.215804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.215774    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks" pod "371d593b-88c7-4bb3-8624-46bed833916b" (UID: "371d593b-88c7-4bb3-8624-46bed833916b"). InnerVolumeSpecName "kube-api-access-4hjks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:35.315011 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.314988    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjks\" (UniqueName: \"kubernetes.io/projected/371d593b-88c7-4bb3-8624-46bed833916b-kube-api-access-4hjks\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:35.744918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.744888    5533 generic.go:356] "Generic (PLEG): container finished" podID="c650e378-4c94-41b2-a05c-e89f4d51ed14" containerID="574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779" exitCode=0
Nov 03 21:52:35.745297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.744931    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"
Nov 03 21:52:35.745297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.744977    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" event={"ID":"c650e378-4c94-41b2-a05c-e89f4d51ed14","Type":"ContainerDied","Data":"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"}
Nov 03 21:52:35.745297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.745040    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr" event={"ID":"c650e378-4c94-41b2-a05c-e89f4d51ed14","Type":"ContainerDied","Data":"f1e7a54abe72b113f66bb39e87174defdf20111f65f144431d61b39b383b604c"}
Nov 03 21:52:35.745297 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.745076    5533 scope.go:122] "RemoveContainer" containerID="574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"
Nov 03 21:52:35.747047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.747024    5533 generic.go:356] "Generic (PLEG): container finished" podID="371d593b-88c7-4bb3-8624-46bed833916b" containerID="31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b" exitCode=0
Nov 03 21:52:35.747189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.747071    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"
Nov 03 21:52:35.747189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.747125    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" event={"ID":"371d593b-88c7-4bb3-8624-46bed833916b","Type":"ContainerDied","Data":"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"}
Nov 03 21:52:35.747189 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.747159    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g" event={"ID":"371d593b-88c7-4bb3-8624-46bed833916b","Type":"ContainerDied","Data":"c9838af0ccb10310d46e36c71030f782b289faa4f30b5cebf81f51ea01e97cea"}
Nov 03 21:52:35.757997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.757895    5533 scope.go:122] "RemoveContainer" containerID="574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"
Nov 03 21:52:35.758320 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:35.758290    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779\": container with ID starting with 574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779 not found: ID does not exist" containerID="574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"
Nov 03 21:52:35.758418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.758330    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779"} err="failed to get container status \"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779\": rpc error: code = NotFound desc = could not find container \"574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779\": container with ID starting with 574d30c53b4cf0414a072ec2908be9d64e7e4580b38b56bc54e32ddf165fc779 not found: ID does not exist"
Nov 03 21:52:35.758418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.758364    5533 scope.go:122] "RemoveContainer" containerID="31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"
Nov 03 21:52:35.767974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.767955    5533 scope.go:122] "RemoveContainer" containerID="31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"
Nov 03 21:52:35.768224 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:35.768199    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b\": container with ID starting with 31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b not found: ID does not exist" containerID="31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"
Nov 03 21:52:35.768323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.768225    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b"} err="failed to get container status \"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b\": rpc error: code = NotFound desc = could not find container \"31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b\": container with ID starting with 31cbf9347e6eb3d3bc51adbaee8a48a1ad456d94da30dfc80c4523ee17594b0b not found: ID does not exist"
Nov 03 21:52:35.901034 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.900963    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:52:35.948163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:35.948136    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-13c6b6-5c69f6b787-zdf9g"]
Nov 03 21:52:36.008078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:36.008013    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:52:36.012681 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:36.012658    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-13c6b6-fbbbf6474-jshxr"]
Nov 03 21:52:36.495718 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:36.495688    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371d593b-88c7-4bb3-8624-46bed833916b" path="/var/lib/kubelet/pods/371d593b-88c7-4bb3-8624-46bed833916b/volumes"
Nov 03 21:52:36.496138 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:36.496124    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c650e378-4c94-41b2-a05c-e89f4d51ed14" path="/var/lib/kubelet/pods/c650e378-4c94-41b2-a05c-e89f4d51ed14/volumes"
Nov 03 21:52:49.265399 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:49.264027    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:52:51.036425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:51.036391    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:52:51.146226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:51.146200    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:52:56.037170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.037120    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" podUID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" containerName="payload-processing" containerID="cri-o://ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c" gracePeriod=25
Nov 03 21:52:56.147377 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.147337    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" podUID="2dfab88c-7b77-4cdd-8870-57a86a702162" containerName="payload-pre-processing" containerID="cri-o://0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9" gracePeriod=25
Nov 03 21:52:56.316904 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.316874    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:52:56.397794 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.397763    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\") pod \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") "
Nov 03 21:52:56.397959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.397800    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\" (UniqueName: \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\") pod \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\" (UID: \"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80\") "
Nov 03 21:52:56.398199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.398173    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume" pod "68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" (UID: "68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:56.398463 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.398440    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.400250 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.400227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5" pod "68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" (UID: "68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80"). InnerVolumeSpecName "kube-api-access-p9vj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:56.430970 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.430666    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:52:56.498855 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.498830    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\") pod \"2dfab88c-7b77-4cdd-8870-57a86a702162\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") "
Nov 03 21:52:56.498999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.498876    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\" (UniqueName: \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\") pod \"2dfab88c-7b77-4cdd-8870-57a86a702162\" (UID: \"2dfab88c-7b77-4cdd-8870-57a86a702162\") "
Nov 03 21:52:56.499269 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.499242    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume" pod "2dfab88c-7b77-4cdd-8870-57a86a702162" (UID: "2dfab88c-7b77-4cdd-8870-57a86a702162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:52:56.499476 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.499457    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dfab88c-7b77-4cdd-8870-57a86a702162-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.499566 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.499481    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vj5\" (UniqueName: \"kubernetes.io/projected/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80-kube-api-access-p9vj5\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.501042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.501017    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps" pod "2dfab88c-7b77-4cdd-8870-57a86a702162" (UID: "2dfab88c-7b77-4cdd-8870-57a86a702162"). InnerVolumeSpecName "kube-api-access-nc6ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:52:56.600732 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.600653    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6ps\" (UniqueName: \"kubernetes.io/projected/2dfab88c-7b77-4cdd-8870-57a86a702162-kube-api-access-nc6ps\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:52:56.907199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.907116    5533 generic.go:356] "Generic (PLEG): container finished" podID="2dfab88c-7b77-4cdd-8870-57a86a702162" containerID="0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9" exitCode=0
Nov 03 21:52:56.907199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.907148    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"
Nov 03 21:52:56.907199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.907184    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" event={"ID":"2dfab88c-7b77-4cdd-8870-57a86a702162","Type":"ContainerDied","Data":"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"}
Nov 03 21:52:56.907496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.907242    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss" event={"ID":"2dfab88c-7b77-4cdd-8870-57a86a702162","Type":"ContainerDied","Data":"71eb214e4ab60d7c9754a50390028dbaa5915144aa6eced4e3c84830d9b2fc1e"}
Nov 03 21:52:56.907496 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.907268    5533 scope.go:122] "RemoveContainer" containerID="0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"
Nov 03 21:52:56.909008 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.908981    5533 generic.go:356] "Generic (PLEG): container finished" podID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" containerID="ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c" exitCode=0
Nov 03 21:52:56.909173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.909033    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"
Nov 03 21:52:56.909173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.909108    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" event={"ID":"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80","Type":"ContainerDied","Data":"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"}
Nov 03 21:52:56.909173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.909145    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc" event={"ID":"68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80","Type":"ContainerDied","Data":"7b9bfbfba6c5d2ef97f30a839906227ad4819964ef80ac7a34c5734802cb020b"}
Nov 03 21:52:56.919239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.919221    5533 scope.go:122] "RemoveContainer" containerID="0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"
Nov 03 21:52:56.919495 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:56.919476    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9\": container with ID starting with 0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9 not found: ID does not exist" containerID="0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"
Nov 03 21:52:56.919545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.919505    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9"} err="failed to get container status \"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9\": rpc error: code = NotFound desc = could not find container \"0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9\": container with ID starting with 0d31ff72b3f738b4ff539cd7c144781a8fc69b1e64b421db48e5c585c7dd76b9 not found: ID does not exist"
Nov 03 21:52:56.919545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.919524    5533 scope.go:122] "RemoveContainer" containerID="ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"
Nov 03 21:52:56.928759 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.928744    5533 scope.go:122] "RemoveContainer" containerID="ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"
Nov 03 21:52:56.929032 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:52:56.929014    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c\": container with ID starting with ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c not found: ID does not exist" containerID="ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"
Nov 03 21:52:56.929104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:56.929040    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c"} err="failed to get container status \"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c\": rpc error: code = NotFound desc = could not find container \"ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c\": container with ID starting with ac9fe81bf10962a1e782df8fd2e4b0169d580bb266fc51304b8660e05c08653c not found: ID does not exist"
Nov 03 21:52:57.067672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:57.067625    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:52:57.074194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:57.073918    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-965fe8-cf6bf5d-mb6pc"]
Nov 03 21:52:57.177127 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:57.177094    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:52:57.184458 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:57.184429    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-965fe8-6f99bf547f-xlvss"]
Nov 03 21:52:58.496542 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:58.496506    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfab88c-7b77-4cdd-8870-57a86a702162" path="/var/lib/kubelet/pods/2dfab88c-7b77-4cdd-8870-57a86a702162/volumes"
Nov 03 21:52:58.496972 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:52:58.496957    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" path="/var/lib/kubelet/pods/68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80/volumes"
Nov 03 21:53:07.180413 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:07.180326    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:53:16.668042 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:16.668004    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:53:16.775206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:16.775171    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:53:18.064490 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.064455    5533 generic.go:356] "Generic (PLEG): container finished" podID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerID="f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd" exitCode=0
Nov 03 21:53:18.065053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.064530    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerDied","Data":"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"}
Nov 03 21:53:18.065053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.065017    5533 scope.go:122] "RemoveContainer" containerID="f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"
Nov 03 21:53:18.066585 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.066558    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:53:18.066686 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.066624    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:53:18.066748 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:18.066691    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:53:19.085832 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.085795    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerStarted","Data":"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"}
Nov 03 21:53:19.214699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.214670    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:53:19.226264 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.226237    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:53:19.283827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.283728    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:53:19.291306 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:53:19.291271    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:53:19.297153 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.297135    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:53:19.438078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.438042    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:53:19.439113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.439086    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:53:19.439247 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:19.439134    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:53:20.438100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:20.438060    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:53:20.438699 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:20.438120    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:53:21.438841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:21.438804    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:53:21.439249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:21.438861    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:53:21.444121 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:21.444091    5533 scope.go:122] "RemoveContainer" containerID="45132dd3879588ac97363540ba0c13903ed8b1a4f850acc553d54838cee0c611"
Nov 03 21:53:21.673394 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:21.672691    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" podUID="13e15500-039d-40f5-a602-b6071eec65a8" containerName="payload-processing" containerID="cri-o://317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671" gracePeriod=25
Nov 03 21:53:21.781660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:21.780854    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" podUID="6552992b-bd0a-4449-a504-cece808d68e3" containerName="payload-pre-processing" containerID="cri-o://2448a5905ea679a16e24cecec51febff363485614f1ff0db64fc8515f7c0b353" gracePeriod=25
Nov 03 21:53:21.929576 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:53:21.929550    5533 logging.go:55] [core] [Channel #14632 SubChannel #14634]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.24:50052", ServerName: "10.133.1.24:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.24:50052: connect: connection refused"
Nov 03 21:53:22.031929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.031896    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:53:22.121143 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.121047    5533 generic.go:356] "Generic (PLEG): container finished" podID="6552992b-bd0a-4449-a504-cece808d68e3" containerID="2448a5905ea679a16e24cecec51febff363485614f1ff0db64fc8515f7c0b353" exitCode=0
Nov 03 21:53:22.121293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.121185    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" event={"ID":"6552992b-bd0a-4449-a504-cece808d68e3","Type":"ContainerDied","Data":"2448a5905ea679a16e24cecec51febff363485614f1ff0db64fc8515f7c0b353"}
Nov 03 21:53:22.121293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.121241    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" event={"ID":"6552992b-bd0a-4449-a504-cece808d68e3","Type":"ContainerDied","Data":"f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54"}
Nov 03 21:53:22.121293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.121268    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5baaf61b0d842fb71f3667551af5fdf891d60525b385de88007520ec22ddf54"
Nov 03 21:53:22.124922 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.124900    5533 generic.go:356] "Generic (PLEG): container finished" podID="13e15500-039d-40f5-a602-b6071eec65a8" containerID="317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671" exitCode=0
Nov 03 21:53:22.125044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.125001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" event={"ID":"13e15500-039d-40f5-a602-b6071eec65a8","Type":"ContainerDied","Data":"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"}
Nov 03 21:53:22.125044 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.125033    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" event={"ID":"13e15500-039d-40f5-a602-b6071eec65a8","Type":"ContainerDied","Data":"61ae81b228f7abccc4113df771eafcca1e719fafd47022c50a147b124974397b"}
Nov 03 21:53:22.125193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.125059    5533 scope.go:122] "RemoveContainer" containerID="317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"
Nov 03 21:53:22.125193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.125163    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"
Nov 03 21:53:22.127956 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.127937    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:53:22.131505 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.131484    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\" (UniqueName: \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\") pod \"13e15500-039d-40f5-a602-b6071eec65a8\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") "
Nov 03 21:53:22.131656 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.131625    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\") pod \"13e15500-039d-40f5-a602-b6071eec65a8\" (UID: \"13e15500-039d-40f5-a602-b6071eec65a8\") "
Nov 03 21:53:22.132393 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.132376    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume" pod "13e15500-039d-40f5-a602-b6071eec65a8" (UID: "13e15500-039d-40f5-a602-b6071eec65a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:22.134228 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.134209    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54" pod "13e15500-039d-40f5-a602-b6071eec65a8" (UID: "13e15500-039d-40f5-a602-b6071eec65a8"). InnerVolumeSpecName "kube-api-access-mpm54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:22.136291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.136277    5533 scope.go:122] "RemoveContainer" containerID="317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"
Nov 03 21:53:22.136545 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:53:22.136517    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671\": container with ID starting with 317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671 not found: ID does not exist" containerID="317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"
Nov 03 21:53:22.136592 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.136545    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671"} err="failed to get container status \"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671\": rpc error: code = NotFound desc = could not find container \"317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671\": container with ID starting with 317c37a353c083fa79c5e3e64d3f16b6f2e28c488f57cc7f6ff3accc76f5d671 not found: ID does not exist"
Nov 03 21:53:22.233122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233093    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\") pod \"6552992b-bd0a-4449-a504-cece808d68e3\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") "
Nov 03 21:53:22.233289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233151    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\" (UniqueName: \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\") pod \"6552992b-bd0a-4449-a504-cece808d68e3\" (UID: \"6552992b-bd0a-4449-a504-cece808d68e3\") "
Nov 03 21:53:22.233406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233386    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume" pod "6552992b-bd0a-4449-a504-cece808d68e3" (UID: "6552992b-bd0a-4449-a504-cece808d68e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:22.233902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233878    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6552992b-bd0a-4449-a504-cece808d68e3-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:22.233902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233901    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpm54\" (UniqueName: \"kubernetes.io/projected/13e15500-039d-40f5-a602-b6071eec65a8-kube-api-access-mpm54\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:22.234093 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.233911    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e15500-039d-40f5-a602-b6071eec65a8-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:22.235320 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.235298    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88" pod "6552992b-bd0a-4449-a504-cece808d68e3" (UID: "6552992b-bd0a-4449-a504-cece808d68e3"). InnerVolumeSpecName "kube-api-access-cfz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:22.335530 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.335443    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz88\" (UniqueName: \"kubernetes.io/projected/6552992b-bd0a-4449-a504-cece808d68e3-kube-api-access-cfz88\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:22.438306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.438276    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:53:22.438450 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.438336    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:53:22.541249 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.541215    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:53:22.545150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.545121    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr"]
Nov 03 21:53:22.930164 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.930119    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-267053-58ccd77958-cw6tr" podUID="13e15500-039d-40f5-a602-b6071eec65a8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.25:50052\" within 1s: context deadline exceeded"
Nov 03 21:53:22.930344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:22.930206    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp" podUID="6552992b-bd0a-4449-a504-cece808d68e3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.24:50052\" within 1s: context deadline exceeded"
Nov 03 21:53:22.930344 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:53:22.930247    5533 logging.go:55] [core] [Channel #14631 SubChannel #14633]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.25:50052", ServerName: "10.133.1.25:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.25:50052: operation was canceled"
Nov 03 21:53:22.930344 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:53:22.930277    5533 logging.go:55] [core] [Channel #14632 SubChannel #14634]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.24:50052", ServerName: "10.133.1.24:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.24:50052: operation was canceled"
Nov 03 21:53:23.033775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.033690    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:53:23.133841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.133810    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"
Nov 03 21:53:23.295600 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.295502    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:53:23.298974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.298943    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-267053-8487c77c8c-v5rkp"]
Nov 03 21:53:23.438468 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.438437    5533 patch_prober.go:29] interesting pod/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Nov 03 21:53:23.438647 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:23.438491    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Nov 03 21:53:24.442942 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:24.442913    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:53:24.444128 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:24.444102    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:53:24.494771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:24.494746    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e15500-039d-40f5-a602-b6071eec65a8" path="/var/lib/kubelet/pods/13e15500-039d-40f5-a602-b6071eec65a8/volumes"
Nov 03 21:53:24.495206 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:24.495191    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6552992b-bd0a-4449-a504-cece808d68e3" path="/var/lib/kubelet/pods/6552992b-bd0a-4449-a504-cece808d68e3/volumes"
Nov 03 21:53:49.048454 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:49.048399    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"]
Nov 03 21:53:49.094836 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:49.094808    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae41440-t7xlp"]
Nov 03 21:53:49.261342 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:53:49.261310    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:53:50.494849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:50.494810    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c637588-7799-40c2-a0ea-4e0a6f091c3f" path="/var/lib/kubelet/pods/4c637588-7799-40c2-a0ea-4e0a6f091c3f/volumes"
Nov 03 21:53:53.938649 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.938606    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"]
Nov 03 21:53:53.939345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939323    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2dfab88c-7b77-4cdd-8870-57a86a702162" containerName="payload-pre-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939346    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfab88c-7b77-4cdd-8870-57a86a702162" containerName="payload-pre-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939393    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c650e378-4c94-41b2-a05c-e89f4d51ed14" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939401    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c650e378-4c94-41b2-a05c-e89f4d51ed14" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939425    5533 cpu_manager.go:414] "containerMap: removing container" podUID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939433    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939446    5533 cpu_manager.go:414] "containerMap: removing container" podUID="13e15500-039d-40f5-a602-b6071eec65a8" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939450    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e15500-039d-40f5-a602-b6071eec65a8" containerName="payload-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939461    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6552992b-bd0a-4449-a504-cece808d68e3" containerName="payload-pre-processing"
Nov 03 21:53:53.939465 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939466    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6552992b-bd0a-4449-a504-cece808d68e3" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939476    5533 cpu_manager.go:414] "containerMap: removing container" podUID="371d593b-88c7-4bb3-8624-46bed833916b" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939482    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371d593b-88c7-4bb3-8624-46bed833916b" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939601    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e732f3-e9c1-45d2-b1f3-5f4ecfeb6b80" containerName="payload-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939610    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e15500-039d-40f5-a602-b6071eec65a8" containerName="payload-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939617    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6552992b-bd0a-4449-a504-cece808d68e3" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939623    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="371d593b-88c7-4bb3-8624-46bed833916b" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939644    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfab88c-7b77-4cdd-8870-57a86a702162" containerName="payload-pre-processing"
Nov 03 21:53:53.939818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.939655    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c650e378-4c94-41b2-a05c-e89f4d51ed14" containerName="payload-processing"
Nov 03 21:53:53.944064 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.944046    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:53.997663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:53.997625    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"]
Nov 03 21:53:54.024613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.024584    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tr6l\" (UniqueName: \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.024776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.024713    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.024776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.024749    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.039809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.039784    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:53:54.039809 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.039796    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:53:54.125987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.125959    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tr6l\" (UniqueName: \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.126141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.126021    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.126141 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.126057    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.126788 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.126770    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.128524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.128505    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.183371 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.183346    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tr6l\" (UniqueName: \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\") pod \"maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.258924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.258869    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:54.631266 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:54.631230    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"]
Nov 03 21:53:54.631891 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:53:54.631863    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f44bdf_2410_4a8e_b6fe_9169e5960769.slice/crio-4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3 WatchSource:0}: Error finding container 4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3: Status 404 returned error can't find the container with id 4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3
Nov 03 21:53:55.369582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:55.369502    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx" event={"ID":"74f44bdf-2410-4a8e-b6fe-9169e5960769","Type":"ContainerStarted","Data":"be30da01cea0b5342d7659885e453d6045940d548f9c6f20010b6b1fc7527b36"}
Nov 03 21:53:55.369582 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:55.369561    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx" event={"ID":"74f44bdf-2410-4a8e-b6fe-9169e5960769","Type":"ContainerStarted","Data":"4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3"}
Nov 03 21:53:55.479771 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:55.479716    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx" podStartSLOduration=2.479698822 podStartE2EDuration="2.479698822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:55.478349086 +0000 UTC m=+3937.109662564" watchObservedRunningTime="2026-11-03 21:53:55.479698822 +0000 UTC m=+3937.111012301"
Nov 03 21:53:56.378080 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:56.378043    5533 generic.go:356] "Generic (PLEG): container finished" podID="74f44bdf-2410-4a8e-b6fe-9169e5960769" containerID="be30da01cea0b5342d7659885e453d6045940d548f9c6f20010b6b1fc7527b36" exitCode=0
Nov 03 21:53:56.378502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:56.378132    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx" event={"ID":"74f44bdf-2410-4a8e-b6fe-9169e5960769","Type":"ContainerDied","Data":"be30da01cea0b5342d7659885e453d6045940d548f9c6f20010b6b1fc7527b36"}
Nov 03 21:53:57.535965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.535941    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:57.666695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.666113    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\") pod \"74f44bdf-2410-4a8e-b6fe-9169e5960769\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") "
Nov 03 21:53:57.666695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.666230    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\") pod \"74f44bdf-2410-4a8e-b6fe-9169e5960769\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") "
Nov 03 21:53:57.666695 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.666256    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\" (UniqueName: \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\") pod \"74f44bdf-2410-4a8e-b6fe-9169e5960769\" (UID: \"74f44bdf-2410-4a8e-b6fe-9169e5960769\") "
Nov 03 21:53:57.666989 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.666959    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca" pod "74f44bdf-2410-4a8e-b6fe-9169e5960769" (UID: "74f44bdf-2410-4a8e-b6fe-9169e5960769"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:57.668918 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.668877    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca" pod "74f44bdf-2410-4a8e-b6fe-9169e5960769" (UID: "74f44bdf-2410-4a8e-b6fe-9169e5960769"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:53:57.669199 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.669165    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l" pod "74f44bdf-2410-4a8e-b6fe-9169e5960769" (UID: "74f44bdf-2410-4a8e-b6fe-9169e5960769"). InnerVolumeSpecName "kube-api-access-5tr6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:57.767430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.767398    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:57.767430 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.767426    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tr6l\" (UniqueName: \"kubernetes.io/projected/74f44bdf-2410-4a8e-b6fe-9169e5960769-kube-api-access-5tr6l\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:57.767587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:57.767441    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f44bdf-2410-4a8e-b6fe-9169e5960769-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:53:58.397586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:58.397554    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"
Nov 03 21:53:58.397586 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:58.397583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx" event={"ID":"74f44bdf-2410-4a8e-b6fe-9169e5960769","Type":"ContainerDied","Data":"4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3"}
Nov 03 21:53:58.397837 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:53:58.397651    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d51a50cdc301676633bf80ddc43b5ea9f2a3f132768fb1ed8387d0b313e9fd3"
Nov 03 21:54:04.898738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:04.898705    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:54:19.261622 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:54:19.261588    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:54:21.523176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:21.523138    5533 scope.go:122] "RemoveContainer" containerID="2448a5905ea679a16e24cecec51febff363485614f1ff0db64fc8515f7c0b353"
Nov 03 21:54:30.807974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:30.807931    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:54:30.809204 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:30.808877    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy" containerID="cri-o://c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef" gracePeriod=30
Nov 03 21:54:36.075740 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.075709    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:54:36.234673 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234646    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.234818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234697    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.234818 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234801    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.234953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234839    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.234953 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234876    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.235073 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.234985    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.235160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235079    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:36.235498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235435    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:36.235498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235449    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:36.235716 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235656    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:36.235825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235805    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.235894 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235879    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.236097 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.235939    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\") pod \"87039730-f061-4333-a820-7ba0f7de70da\" (UID: \"87039730-f061-4333-a820-7ba0f7de70da\") "
Nov 03 21:54:36.236214 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236181    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:36.236849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236759    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.236849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236787    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.236849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236802    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.236849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236816    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.236849 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.236829    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87039730-f061-4333-a820-7ba0f7de70da-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.237384 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.237353    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:36.239858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.239781    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "kube-api-access-4vh4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:36.239971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.239880    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:54:36.240104 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.240060    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token" pod "87039730-f061-4333-a820-7ba0f7de70da" (UID: "87039730-f061-4333-a820-7ba0f7de70da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:36.338493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.338465    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.338670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.338520    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vh4t\" (UniqueName: \"kubernetes.io/projected/87039730-f061-4333-a820-7ba0f7de70da-kube-api-access-4vh4t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.338670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.338536    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87039730-f061-4333-a820-7ba0f7de70da-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.338670 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.338548    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87039730-f061-4333-a820-7ba0f7de70da-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:36.804536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.804454    5533 generic.go:356] "Generic (PLEG): container finished" podID="87039730-f061-4333-a820-7ba0f7de70da" containerID="c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef" exitCode=0
Nov 03 21:54:36.804536 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.804515    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"
Nov 03 21:54:36.804803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.804544    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerDied","Data":"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"}
Nov 03 21:54:36.804803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.804602    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z" event={"ID":"87039730-f061-4333-a820-7ba0f7de70da","Type":"ContainerDied","Data":"7548c37539f57621593533d03764150c9ad9e3e458840d0e2177abe4b8fb4b17"}
Nov 03 21:54:36.804803 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.804630    5533 scope.go:122] "RemoveContainer" containerID="c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"
Nov 03 21:54:36.814998 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.814980    5533 scope.go:122] "RemoveContainer" containerID="b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"
Nov 03 21:54:36.824159 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.824142    5533 scope.go:122] "RemoveContainer" containerID="c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"
Nov 03 21:54:36.824439 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:54:36.824418    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef\": container with ID starting with c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef not found: ID does not exist" containerID="c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"
Nov 03 21:54:36.824543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.824449    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef"} err="failed to get container status \"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef\": rpc error: code = NotFound desc = could not find container \"c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef\": container with ID starting with c0cd36ab044be764e808556c538905489b788433fa0f43465032e5eddf2d93ef not found: ID does not exist"
Nov 03 21:54:36.824543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.824472    5533 scope.go:122] "RemoveContainer" containerID="b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"
Nov 03 21:54:36.824727 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:54:36.824710    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd\": container with ID starting with b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd not found: ID does not exist" containerID="b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"
Nov 03 21:54:36.824776 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:36.824734    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd"} err="failed to get container status \"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd\": rpc error: code = NotFound desc = could not find container \"b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd\": container with ID starting with b2984522e3e2e85a7ed1f813fa51577226a4a5782a6927f4676ac2e8d4e3e9cd not found: ID does not exist"
Nov 03 21:54:37.008727 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:37.008700    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:54:37.054708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:37.054618    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-267053-openshift-default-7cc6f6666f-wtk7z"]
Nov 03 21:54:38.280202 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:38.280166    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:54:38.493996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:38.493963    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87039730-f061-4333-a820-7ba0f7de70da" path="/var/lib/kubelet/pods/87039730-f061-4333-a820-7ba0f7de70da/volumes"
Nov 03 21:54:43.280895 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.280847    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" podUID="5b58f63d-506b-4277-8948-4fff8fc3869e" containerName="payload-processing" containerID="cri-o://b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef" gracePeriod=25
Nov 03 21:54:43.559702 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.559678    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:54:43.627339 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.627311    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\") pod \"5b58f63d-506b-4277-8948-4fff8fc3869e\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") "
Nov 03 21:54:43.627631 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.627610    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume" pod "5b58f63d-506b-4277-8948-4fff8fc3869e" (UID: "5b58f63d-506b-4277-8948-4fff8fc3869e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:43.728556 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.728528    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\" (UniqueName: \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\") pod \"5b58f63d-506b-4277-8948-4fff8fc3869e\" (UID: \"5b58f63d-506b-4277-8948-4fff8fc3869e\") "
Nov 03 21:54:43.728915 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.728892    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b58f63d-506b-4277-8948-4fff8fc3869e-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:43.730526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.730502    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq" pod "5b58f63d-506b-4277-8948-4fff8fc3869e" (UID: "5b58f63d-506b-4277-8948-4fff8fc3869e"). InnerVolumeSpecName "kube-api-access-4sphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:43.829658 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.829570    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sphq\" (UniqueName: \"kubernetes.io/projected/5b58f63d-506b-4277-8948-4fff8fc3869e-kube-api-access-4sphq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:54:43.856442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.856337    5533 generic.go:356] "Generic (PLEG): container finished" podID="5b58f63d-506b-4277-8948-4fff8fc3869e" containerID="b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef" exitCode=0
Nov 03 21:54:43.856442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.856373    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" event={"ID":"5b58f63d-506b-4277-8948-4fff8fc3869e","Type":"ContainerDied","Data":"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"}
Nov 03 21:54:43.856442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.856412    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"
Nov 03 21:54:43.856442 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.856432    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2" event={"ID":"5b58f63d-506b-4277-8948-4fff8fc3869e","Type":"ContainerDied","Data":"b7a54ebccf91e0cee81e6cf0031a78c88b04a5eef832acfcd93825790e7e37a0"}
Nov 03 21:54:43.856758 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.856455    5533 scope.go:122] "RemoveContainer" containerID="b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"
Nov 03 21:54:43.867144 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.867124    5533 scope.go:122] "RemoveContainer" containerID="b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"
Nov 03 21:54:43.867418 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:54:43.867390    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef\": container with ID starting with b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef not found: ID does not exist" containerID="b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"
Nov 03 21:54:43.867493 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:43.867422    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef"} err="failed to get container status \"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef\": rpc error: code = NotFound desc = could not find container \"b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef\": container with ID starting with b1726442e95d433a8bb969f88dbff4a98df1558b53fa96892c6b0011757e2aef not found: ID does not exist"
Nov 03 21:54:44.012338 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:44.012314    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:54:44.016555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:44.016532    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-76eaf8-7b64d7459b-49tw2"]
Nov 03 21:54:44.496239 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:54:44.496194    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b58f63d-506b-4277-8948-4fff8fc3869e" path="/var/lib/kubelet/pods/5b58f63d-506b-4277-8948-4fff8fc3869e/volumes"
Nov 03 21:54:49.260854 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:54:49.260821    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:55:06.700723 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:06.700690    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:55:06.809329 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:06.809296    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:55:11.701929 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:11.701888    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" podUID="1e68e560-0363-4b83-9b6f-702cf5644ffc" containerName="payload-processing" containerID="cri-o://6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7" gracePeriod=25
Nov 03 21:55:11.811437 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:11.811396    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" podUID="0721d28b-4d5c-4772-9434-dba80a2cd05c" containerName="payload-pre-processing" containerID="cri-o://fcc74106e9a4e428c1d9e509faafd4eb5f268b504d1faa17d7d3c6ac10c32555" gracePeriod=25
Nov 03 21:55:11.990184 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:11.989823    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:55:12.055053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.055015    5533 generic.go:356] "Generic (PLEG): container finished" podID="0721d28b-4d5c-4772-9434-dba80a2cd05c" containerID="fcc74106e9a4e428c1d9e509faafd4eb5f268b504d1faa17d7d3c6ac10c32555" exitCode=0
Nov 03 21:55:12.055240 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.055126    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" event={"ID":"0721d28b-4d5c-4772-9434-dba80a2cd05c","Type":"ContainerDied","Data":"fcc74106e9a4e428c1d9e509faafd4eb5f268b504d1faa17d7d3c6ac10c32555"}
Nov 03 21:55:12.056846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.056821    5533 generic.go:356] "Generic (PLEG): container finished" podID="1e68e560-0363-4b83-9b6f-702cf5644ffc" containerID="6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7" exitCode=0
Nov 03 21:55:12.057001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.056874    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" event={"ID":"1e68e560-0363-4b83-9b6f-702cf5644ffc","Type":"ContainerDied","Data":"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"}
Nov 03 21:55:12.057001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.056907    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2" event={"ID":"1e68e560-0363-4b83-9b6f-702cf5644ffc","Type":"ContainerDied","Data":"c7d6b5728e253e3e73449d541cdd3f7a8c13dae4d633a40044118e9cc0a6441d"}
Nov 03 21:55:12.057001 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.056928    5533 scope.go:122] "RemoveContainer" containerID="6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"
Nov 03 21:55:12.057187 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.057058    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"
Nov 03 21:55:12.069293 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.069274    5533 scope.go:122] "RemoveContainer" containerID="6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"
Nov 03 21:55:12.069573 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:12.069554    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7\": container with ID starting with 6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7 not found: ID does not exist" containerID="6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"
Nov 03 21:55:12.069685 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.069578    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7"} err="failed to get container status \"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7\": rpc error: code = NotFound desc = could not find container \"6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7\": container with ID starting with 6254d4cea927c83c3a01c32f205393bd5c6c394176c690f7a87dad2d10a906a7 not found: ID does not exist"
Nov 03 21:55:12.100920 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.100891    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:55:12.148381 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.148353    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\") pod \"1e68e560-0363-4b83-9b6f-702cf5644ffc\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") "
Nov 03 21:55:12.148544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.148532    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\") pod \"1e68e560-0363-4b83-9b6f-702cf5644ffc\" (UID: \"1e68e560-0363-4b83-9b6f-702cf5644ffc\") "
Nov 03 21:55:12.148808 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.148781    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume" pod "1e68e560-0363-4b83-9b6f-702cf5644ffc" (UID: "1e68e560-0363-4b83-9b6f-702cf5644ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:12.149037 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.149019    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68e560-0363-4b83-9b6f-702cf5644ffc-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:12.150535 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.150517    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs" pod "1e68e560-0363-4b83-9b6f-702cf5644ffc" (UID: "1e68e560-0363-4b83-9b6f-702cf5644ffc"). InnerVolumeSpecName "kube-api-access-sdqcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:12.249429 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.249362    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\") pod \"0721d28b-4d5c-4772-9434-dba80a2cd05c\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") "
Nov 03 21:55:12.249546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.249431    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\" (UniqueName: \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\") pod \"0721d28b-4d5c-4772-9434-dba80a2cd05c\" (UID: \"0721d28b-4d5c-4772-9434-dba80a2cd05c\") "
Nov 03 21:55:12.249713 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.249692    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume" pod "0721d28b-4d5c-4772-9434-dba80a2cd05c" (UID: "0721d28b-4d5c-4772-9434-dba80a2cd05c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:12.249924 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.249909    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/1e68e560-0363-4b83-9b6f-702cf5644ffc-kube-api-access-sdqcs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:12.249978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.249926    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0721d28b-4d5c-4772-9434-dba80a2cd05c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:12.251596 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.251572    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv" pod "0721d28b-4d5c-4772-9434-dba80a2cd05c" (UID: "0721d28b-4d5c-4772-9434-dba80a2cd05c"). InnerVolumeSpecName "kube-api-access-bm9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:12.351509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.351473    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9gv\" (UniqueName: \"kubernetes.io/projected/0721d28b-4d5c-4772-9434-dba80a2cd05c-kube-api-access-bm9gv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:12.475734 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.475688    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:55:12.480573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.480548    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c432-68b898c668-dwxq2"]
Nov 03 21:55:12.497113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:12.497082    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e68e560-0363-4b83-9b6f-702cf5644ffc" path="/var/lib/kubelet/pods/1e68e560-0363-4b83-9b6f-702cf5644ffc/volumes"
Nov 03 21:55:13.067155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.067122    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt" event={"ID":"0721d28b-4d5c-4772-9434-dba80a2cd05c","Type":"ContainerDied","Data":"5fcf439ae725f19bf36cb20b802636272e2d9d72475124135775a1dad35e9fe3"}
Nov 03 21:55:13.067155 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.067148    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"
Nov 03 21:55:13.067747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.067182    5533 scope.go:122] "RemoveContainer" containerID="fcc74106e9a4e428c1d9e509faafd4eb5f268b504d1faa17d7d3c6ac10c32555"
Nov 03 21:55:13.231745 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.231710    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:55:13.259302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.259270    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:55:13.263110 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:13.263077    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c432-8cbd6b467-rfwlt"]
Nov 03 21:55:14.493899 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:14.493867    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0721d28b-4d5c-4772-9434-dba80a2cd05c" path="/var/lib/kubelet/pods/0721d28b-4d5c-4772-9434-dba80a2cd05c/volumes"
Nov 03 21:55:19.261706 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:19.261667    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:55:21.562580 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.562550    5533 scope.go:122] "RemoveContainer" containerID="70e49f0d4ff1bbe84ee918b378db9f83cb684b00bafd66d6ec25daf6da8d5f29"
Nov 03 21:55:21.607932 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.607903    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"]
Nov 03 21:55:21.608503 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608486    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5b58f63d-506b-4277-8948-4fff8fc3869e" containerName="payload-processing"
Nov 03 21:55:21.608567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608506    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b58f63d-506b-4277-8948-4fff8fc3869e" containerName="payload-processing"
Nov 03 21:55:21.608567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608532    5533 cpu_manager.go:414] "containerMap: removing container" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608540    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608551    5533 cpu_manager.go:414] "containerMap: removing container" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608556    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608575    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1e68e560-0363-4b83-9b6f-702cf5644ffc" containerName="payload-processing"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608582    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e68e560-0363-4b83-9b6f-702cf5644ffc" containerName="payload-processing"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608597    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0721d28b-4d5c-4772-9434-dba80a2cd05c" containerName="payload-pre-processing"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608605    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0721d28b-4d5c-4772-9434-dba80a2cd05c" containerName="payload-pre-processing"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608623    5533 cpu_manager.go:414] "containerMap: removing container" podUID="74f44bdf-2410-4a8e-b6fe-9169e5960769" containerName="revoke-keys"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608628    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f44bdf-2410-4a8e-b6fe-9169e5960769" containerName="revoke-keys"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608754    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608764    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="87039730-f061-4333-a820-7ba0f7de70da" containerName="istio-proxy"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608771    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e68e560-0363-4b83-9b6f-702cf5644ffc" containerName="payload-processing"
Nov 03 21:55:21.608782 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608780    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0721d28b-4d5c-4772-9434-dba80a2cd05c" containerName="payload-pre-processing"
Nov 03 21:55:21.609091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608788    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f44bdf-2410-4a8e-b6fe-9169e5960769" containerName="revoke-keys"
Nov 03 21:55:21.609091 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.608796    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b58f63d-506b-4277-8948-4fff8fc3869e" containerName="payload-processing"
Nov 03 21:55:21.614161 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.614144    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.649066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.649033    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.649282 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.649265    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.649363 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.649348    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.666529 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.666502    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"]
Nov 03 21:55:21.708212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.708191    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:55:21.708212 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.708207    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:55:21.751124 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.751100    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.751256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.751222    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.751340 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.751265    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.751964 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.751946    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.753524 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.753490    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.810760 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.810733    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\") pod \"maas-api-revoke-sub-b0944058-s96d2\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:21.928409 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:21.928387    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:22.296354 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:22.296321    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"]
Nov 03 21:55:22.298659 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:55:22.298611    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a029610_2b74_42f2_8076_b6034c24a570.slice/crio-7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845 WatchSource:0}: Error finding container 7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845: Status 404 returned error can't find the container with id 7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845
Nov 03 21:55:23.179203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:23.179160    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2" event={"ID":"0a029610-2b74-42f2-8076-b6034c24a570","Type":"ContainerStarted","Data":"35ef6d395e426f7aa4ebada67ee96251575e70aa6c2620437a9bbbf86fca9a78"}
Nov 03 21:55:23.179791 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:23.179229    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2" event={"ID":"0a029610-2b74-42f2-8076-b6034c24a570","Type":"ContainerStarted","Data":"7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845"}
Nov 03 21:55:23.290078 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:23.290033    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2" podStartSLOduration=2.290020289 podStartE2EDuration="2.290020289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:23.288568704 +0000 UTC m=+4024.919882181" watchObservedRunningTime="2026-11-03 21:55:23.290020289 +0000 UTC m=+4024.921333794"
Nov 03 21:55:24.187482 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:24.187451    5533 generic.go:356] "Generic (PLEG): container finished" podID="0a029610-2b74-42f2-8076-b6034c24a570" containerID="35ef6d395e426f7aa4ebada67ee96251575e70aa6c2620437a9bbbf86fca9a78" exitCode=0
Nov 03 21:55:24.187919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:24.187536    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2" event={"ID":"0a029610-2b74-42f2-8076-b6034c24a570","Type":"ContainerDied","Data":"35ef6d395e426f7aa4ebada67ee96251575e70aa6c2620437a9bbbf86fca9a78"}
Nov 03 21:55:24.573965 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:24.573865    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"]
Nov 03 21:55:24.577208 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:24.577186    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-13c6b6-g4sqx"]
Nov 03 21:55:24.818425 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:24.818390    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:55:25.194547 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.194512    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy" containerID="cri-o://c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8" gracePeriod=30
Nov 03 21:55:25.344081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.344058    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:25.412366 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.412343    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\") pod \"0a029610-2b74-42f2-8076-b6034c24a570\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") "
Nov 03 21:55:25.412509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.412463    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\") pod \"0a029610-2b74-42f2-8076-b6034c24a570\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") "
Nov 03 21:55:25.412509 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.412505    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\") pod \"0a029610-2b74-42f2-8076-b6034c24a570\" (UID: \"0a029610-2b74-42f2-8076-b6034c24a570\") "
Nov 03 21:55:25.412987 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.412959    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca" pod "0a029610-2b74-42f2-8076-b6034c24a570" (UID: "0a029610-2b74-42f2-8076-b6034c24a570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:25.414409 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.414379    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w" pod "0a029610-2b74-42f2-8076-b6034c24a570" (UID: "0a029610-2b74-42f2-8076-b6034c24a570"). InnerVolumeSpecName "kube-api-access-cgr8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:25.414502 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.414468    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca" pod "0a029610-2b74-42f2-8076-b6034c24a570" (UID: "0a029610-2b74-42f2-8076-b6034c24a570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:55:25.514258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.514172    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:25.514258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.514203    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a029610-2b74-42f2-8076-b6034c24a570-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:25.514258 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:25.514217    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/0a029610-2b74-42f2-8076-b6034c24a570-kube-api-access-cgr8w\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:26.205337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:26.205303    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"
Nov 03 21:55:26.205845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:26.205297    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2" event={"ID":"0a029610-2b74-42f2-8076-b6034c24a570","Type":"ContainerDied","Data":"7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845"}
Nov 03 21:55:26.205845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:26.205428    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b5b2844dd04e31f7121d5d0220061b892211843c74ce341fbca98e16f0f3845"
Nov 03 21:55:26.494501 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:26.494427    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f44bdf-2410-4a8e-b6fe-9169e5960769" path="/var/lib/kubelet/pods/74f44bdf-2410-4a8e-b6fe-9169e5960769/volumes"
Nov 03 21:55:30.488245 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.488219    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:55:30.567603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567576    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.567763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567667    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.567763 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567732    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.567900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567771    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.567900 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567852    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.568007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567936    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.568007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.567979    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.568120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568011    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.568120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568042    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:30.568120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568064    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:30.568120 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568072    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\") pod \"ae2aed43-c8de-4517-9b6b-051f83195ef2\" (UID: \"ae2aed43-c8de-4517-9b6b-051f83195ef2\") "
Nov 03 21:55:30.568337 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568269    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:30.568587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568557    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:30.568804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568778    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.568888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568811    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.568888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568827    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.568888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.568842    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2aed43-c8de-4517-9b6b-051f83195ef2-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.569150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.569072    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:30.570477 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.570449    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:55:30.570983 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.570956    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "kube-api-access-58qzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:30.571122 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.571087    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:30.571527 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.571507    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy" pod "ae2aed43-c8de-4517-9b6b-051f83195ef2" (UID: "ae2aed43-c8de-4517-9b6b-051f83195ef2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:30.670304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.670271    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.670304 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.670305    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.670520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.670317    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qzv\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-kube-api-access-58qzv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.670520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.670327    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.670520 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:30.670336    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2aed43-c8de-4517-9b6b-051f83195ef2-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:31.241218 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.241184    5533 generic.go:356] "Generic (PLEG): container finished" podID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerID="c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8" exitCode=0
Nov 03 21:55:31.241387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.241231    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerDied","Data":"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"}
Nov 03 21:55:31.241387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.241234    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"
Nov 03 21:55:31.241387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.241280    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j" event={"ID":"ae2aed43-c8de-4517-9b6b-051f83195ef2","Type":"ContainerDied","Data":"62b2c03925bd5740bab8abc7b44843f87ba8f5122c94fcaae4af186aab27b268"}
Nov 03 21:55:31.241387 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.241303    5533 scope.go:122] "RemoveContainer" containerID="c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"
Nov 03 21:55:31.252355 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.252334    5533 scope.go:122] "RemoveContainer" containerID="35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"
Nov 03 21:55:31.261628 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.261602    5533 scope.go:122] "RemoveContainer" containerID="c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"
Nov 03 21:55:31.261934 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:31.261913    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8\": container with ID starting with c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8 not found: ID does not exist" containerID="c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"
Nov 03 21:55:31.262010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.261940    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8"} err="failed to get container status \"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8\": rpc error: code = NotFound desc = could not find container \"c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8\": container with ID starting with c99534c68840c51103c8d411e3364f0659b557a565f65d13138b3d20e4187eb8 not found: ID does not exist"
Nov 03 21:55:31.262010 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.261959    5533 scope.go:122] "RemoveContainer" containerID="35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"
Nov 03 21:55:31.262229 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:31.262209    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f\": container with ID starting with 35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f not found: ID does not exist" containerID="35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"
Nov 03 21:55:31.262302 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.262233    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f"} err="failed to get container status \"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f\": rpc error: code = NotFound desc = could not find container \"35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f\": container with ID starting with 35f3e458696073535723586e8d8bd5eb042662bc45c05b1fe409d82db8d7aa8f not found: ID does not exist"
Nov 03 21:55:31.444616 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.444570    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:55:31.493470 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:31.493383    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-13c6b6-openshift-default-687b769c77-t9l4j"]
Nov 03 21:55:32.494737 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:32.494696    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" path="/var/lib/kubelet/pods/ae2aed43-c8de-4517-9b6b-051f83195ef2/volumes"
Nov 03 21:55:40.696336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:40.696302    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:55:40.696971 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:40.696593    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy" containerID="cri-o://a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0" gracePeriod=30
Nov 03 21:55:45.964100 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:45.964040    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:55:46.048455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048424    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048542    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048615 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048591    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048659    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048742 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048696    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048743    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048844 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048771    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048875    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048922    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\") pod \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\" (UID: \"dc5c70e4-1df2-46e9-93c8-527e004f7aa8\") "
Nov 03 21:55:46.048959 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048942    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:46.049133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.048953    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:46.049133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049004    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:46.049133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:46.049133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049112    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:46.049428 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049405    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.049492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049433    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.049492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049450    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.049492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049464    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.049492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.049478    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.050845 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.050824    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "kube-api-access-xks9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:46.051133 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.051105    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:55:46.051752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.051727    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:46.051752 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.051745    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo" pod "dc5c70e4-1df2-46e9-93c8-527e004f7aa8" (UID: "dc5c70e4-1df2-46e9-93c8-527e004f7aa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:55:46.150565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.150535    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xks9t\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-kube-api-access-xks9t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.150565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.150560    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.150565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.150570    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.150840 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.150579    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5c70e4-1df2-46e9-93c8-527e004f7aa8-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:46.355976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.355904    5533 generic.go:356] "Generic (PLEG): container finished" podID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerID="a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0" exitCode=0
Nov 03 21:55:46.355976 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.355962    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"
Nov 03 21:55:46.356173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.355980    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerDied","Data":"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"}
Nov 03 21:55:46.356173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.356039    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp" event={"ID":"dc5c70e4-1df2-46e9-93c8-527e004f7aa8","Type":"ContainerDied","Data":"fe89e4d0f018b8313cc0d23fea35cab6a5ddda160bae62966014ed2422eee27e"}
Nov 03 21:55:46.356173 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.356067    5533 scope.go:122] "RemoveContainer" containerID="a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"
Nov 03 21:55:46.372017 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.372003    5533 scope.go:122] "RemoveContainer" containerID="f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"
Nov 03 21:55:46.382324 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.382305    5533 scope.go:122] "RemoveContainer" containerID="a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"
Nov 03 21:55:46.382599 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:46.382581    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0\": container with ID starting with a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0 not found: ID does not exist" containerID="a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"
Nov 03 21:55:46.382720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.382605    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0"} err="failed to get container status \"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0\": rpc error: code = NotFound desc = could not find container \"a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0\": container with ID starting with a0b876ad45a460ac5fbce428cfd75e0a08af24d46f5783d16c024f27fb59f9d0 not found: ID does not exist"
Nov 03 21:55:46.382720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.382622    5533 scope.go:122] "RemoveContainer" containerID="f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"
Nov 03 21:55:46.382922 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:46.382899    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd\": container with ID starting with f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd not found: ID does not exist" containerID="f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"
Nov 03 21:55:46.382988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.382927    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd"} err="failed to get container status \"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd\": rpc error: code = NotFound desc = could not find container \"f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd\": container with ID starting with f16c6f543f11a11d66ee367d74988145e0d469f66d633b5428fdfa1d62817ebd not found: ID does not exist"
Nov 03 21:55:46.512434 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.512401    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:55:46.515664 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.515625    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e9142-openshift-default-68c7868db7-zbxqp"]
Nov 03 21:55:46.853361 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.853329    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"]
Nov 03 21:55:46.854007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.853971    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0a029610-2b74-42f2-8076-b6034c24a570" containerName="revoke-keys"
Nov 03 21:55:46.854007 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.853994    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a029610-2b74-42f2-8076-b6034c24a570" containerName="revoke-keys"
Nov 03 21:55:46.854238 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854213    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:55:46.854300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854239    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:55:46.854300 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854292    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:55:46.854374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854301    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:55:46.854374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854318    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854323    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854335    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854340    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854484    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:55:46.854526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854493    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854499    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" containerName="istio-proxy"
Nov 03 21:55:46.854526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.854507    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a029610-2b74-42f2-8076-b6034c24a570" containerName="revoke-keys"
Nov 03 21:55:46.859341 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.859321    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:46.908672 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.908619    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:55:46.908863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.908832    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:55:46.912176 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.912132    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"]
Nov 03 21:55:46.959031 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.959002    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:46.959136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.959040    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:46.959136 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:46.959059    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zmm\" (UniqueName: \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.060708 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.060679    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.061148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.060740    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.061148 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.060767    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zmm\" (UniqueName: \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.061373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.061335    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.063281 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.063261    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.118597 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.118541    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zmm\" (UniqueName: \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\") pod \"maas-api-revoke-sub-ecb4e1a3-xq4bb\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.176958 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.176937    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:47.548735 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:47.548695    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"]
Nov 03 21:55:48.373751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:48.373665    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb" event={"ID":"b3794b91-e9a3-4d98-bf47-a0003d7b0206","Type":"ContainerStarted","Data":"2d9aead31e4e86314ca6a3b02a3341bbadf4debc6485677506430e0c541e192d"}
Nov 03 21:55:48.373751 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:48.373723    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb" event={"ID":"b3794b91-e9a3-4d98-bf47-a0003d7b0206","Type":"ContainerStarted","Data":"0e245d9593a520b200cff8544fe0ed0b42c1788cc3171fbda240a1088c85f2cf"}
Nov 03 21:55:48.484754 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:48.484707    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb" podStartSLOduration=2.484694833 podStartE2EDuration="2.484694833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:48.482911091 +0000 UTC m=+4050.114224568" watchObservedRunningTime="2026-11-03 21:55:48.484694833 +0000 UTC m=+4050.116008330"
Nov 03 21:55:48.493948 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:48.493900    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5c70e4-1df2-46e9-93c8-527e004f7aa8" path="/var/lib/kubelet/pods/dc5c70e4-1df2-46e9-93c8-527e004f7aa8/volumes"
Nov 03 21:55:49.260982 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:55:49.260948    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:55:49.393704 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:49.393675    5533 generic.go:356] "Generic (PLEG): container finished" podID="b3794b91-e9a3-4d98-bf47-a0003d7b0206" containerID="2d9aead31e4e86314ca6a3b02a3341bbadf4debc6485677506430e0c541e192d" exitCode=0
Nov 03 21:55:49.394235 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:49.393779    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb" event={"ID":"b3794b91-e9a3-4d98-bf47-a0003d7b0206","Type":"ContainerDied","Data":"2d9aead31e4e86314ca6a3b02a3341bbadf4debc6485677506430e0c541e192d"}
Nov 03 21:55:50.550053 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.550030    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:50.602373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.602348    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\") pod \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") "
Nov 03 21:55:50.602519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.602418    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\") pod \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") "
Nov 03 21:55:50.602519 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.602449    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\" (UniqueName: \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\") pod \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\" (UID: \"b3794b91-e9a3-4d98-bf47-a0003d7b0206\") "
Nov 03 21:55:50.602954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.602931    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca" pod "b3794b91-e9a3-4d98-bf47-a0003d7b0206" (UID: "b3794b91-e9a3-4d98-bf47-a0003d7b0206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:50.604560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.604528    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm" pod "b3794b91-e9a3-4d98-bf47-a0003d7b0206" (UID: "b3794b91-e9a3-4d98-bf47-a0003d7b0206"). InnerVolumeSpecName "kube-api-access-w6zmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:50.604560 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.604536    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca" pod "b3794b91-e9a3-4d98-bf47-a0003d7b0206" (UID: "b3794b91-e9a3-4d98-bf47-a0003d7b0206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:55:50.703825 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.703794    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:50.703981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.703828    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3794b91-e9a3-4d98-bf47-a0003d7b0206-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:50.703981 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:50.703845    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zmm\" (UniqueName: \"kubernetes.io/projected/b3794b91-e9a3-4d98-bf47-a0003d7b0206-kube-api-access-w6zmm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:55:51.410256 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:51.410226    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"
Nov 03 21:55:51.410827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:51.410227    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb" event={"ID":"b3794b91-e9a3-4d98-bf47-a0003d7b0206","Type":"ContainerDied","Data":"0e245d9593a520b200cff8544fe0ed0b42c1788cc3171fbda240a1088c85f2cf"}
Nov 03 21:55:51.410827 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:55:51.410353    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e245d9593a520b200cff8544fe0ed0b42c1788cc3171fbda240a1088c85f2cf"
Nov 03 21:56:10.826068 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:10.825978    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:56:14.766660 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:14.756481    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 21:56:16.645242 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:16.645208    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:56:16.645741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:16.645465    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy" containerID="cri-o://7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852" gracePeriod=30
Nov 03 21:56:19.261524 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:19.261486    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:56:21.911750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:21.911724    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:56:22.015974 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.015948    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016020    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016062    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016111 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016085    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016146    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016261 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016202    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016266    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:22.016374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016293    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016374 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016324    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016374    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\") pod \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\" (UID: \"cd96c5b1-1787-4cc6-b38e-d9eec976a039\") "
Nov 03 21:56:22.016554 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016361    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:22.016701 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016678    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:22.016747 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.016713    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:22.017062 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.017035    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.017211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.017098    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.017211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.017116    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.017211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.017131    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.017211 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.017169    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:22.018406 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.018379    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:56:22.018525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.018448    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:56:22.018525 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.018503    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:22.018611 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.018545    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx" pod "cd96c5b1-1787-4cc6-b38e-d9eec976a039" (UID: "cd96c5b1-1787-4cc6-b38e-d9eec976a039"). InnerVolumeSpecName "kube-api-access-tn5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:22.118541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.118508    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.118541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.118541    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn5dx\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-kube-api-access-tn5dx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.118743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.118559    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.118743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.118575    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd96c5b1-1787-4cc6-b38e-d9eec976a039-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.118743 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.118589    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd96c5b1-1787-4cc6-b38e-d9eec976a039-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:22.730846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.730811    5533 generic.go:356] "Generic (PLEG): container finished" podID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerID="7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852" exitCode=0
Nov 03 21:56:22.731092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.730862    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"
Nov 03 21:56:22.731092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.730853    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerDied","Data":"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"}
Nov 03 21:56:22.731092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.731002    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns" event={"ID":"cd96c5b1-1787-4cc6-b38e-d9eec976a039","Type":"ContainerDied","Data":"1b1a749269ca683ba633676406dfde7cc138852605af289478e21ee7bb03a73b"}
Nov 03 21:56:22.731092 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.731037    5533 scope.go:122] "RemoveContainer" containerID="7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"
Nov 03 21:56:22.741789 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.741773    5533 scope.go:122] "RemoveContainer" containerID="33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"
Nov 03 21:56:22.750841 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.750816    5533 scope.go:122] "RemoveContainer" containerID="7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"
Nov 03 21:56:22.751168 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:22.751143    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852\": container with ID starting with 7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852 not found: ID does not exist" containerID="7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"
Nov 03 21:56:22.751265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.751173    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852"} err="failed to get container status \"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852\": rpc error: code = NotFound desc = could not find container \"7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852\": container with ID starting with 7563daa1fc2d5c95b9dc6e23298ea218fb492313d2acc1e551afcf84da8d5852 not found: ID does not exist"
Nov 03 21:56:22.751265 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.751191    5533 scope.go:122] "RemoveContainer" containerID="33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"
Nov 03 21:56:22.751511 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:22.751485    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292\": container with ID starting with 33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292 not found: ID does not exist" containerID="33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"
Nov 03 21:56:22.751573 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.751519    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292"} err="failed to get container status \"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292\": rpc error: code = NotFound desc = could not find container \"33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292\": container with ID starting with 33a8b666c0d96095e9c5cad6706f8c0239abb0587965a2b86f2a952a6bbd2292 not found: ID does not exist"
Nov 03 21:56:22.886826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.886743    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:56:22.933113 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:22.933087    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7c432-openshift-default-68d8d9ffb7-k6pns"]
Nov 03 21:56:23.985902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:23.985870    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:56:24.095601 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:24.095569    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:56:24.494464 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:24.494430    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" path="/var/lib/kubelet/pods/cd96c5b1-1787-4cc6-b38e-d9eec976a039/volumes"
Nov 03 21:56:28.986368 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:28.986328    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" podUID="f930e11a-c9d6-40b4-b89f-c266e337addf" containerName="payload-processing" containerID="cri-o://2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f" gracePeriod=25
Nov 03 21:56:29.096244 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.096200    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" podUID="5f94dfa1-639f-4e4d-8104-64059edec661" containerName="payload-pre-processing" containerID="cri-o://12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40" gracePeriod=25
Nov 03 21:56:29.261193 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.261161    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:56:29.379545 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.379519    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:56:29.407382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.407353    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\") pod \"f930e11a-c9d6-40b4-b89f-c266e337addf\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") "
Nov 03 21:56:29.407587 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.407565    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\" (UniqueName: \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\") pod \"f930e11a-c9d6-40b4-b89f-c266e337addf\" (UID: \"f930e11a-c9d6-40b4-b89f-c266e337addf\") "
Nov 03 21:56:29.407804 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.407781    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume" pod "f930e11a-c9d6-40b4-b89f-c266e337addf" (UID: "f930e11a-c9d6-40b4-b89f-c266e337addf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:29.408038 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.408022    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f930e11a-c9d6-40b4-b89f-c266e337addf-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.409671 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.409623    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj" pod "f930e11a-c9d6-40b4-b89f-c266e337addf" (UID: "f930e11a-c9d6-40b4-b89f-c266e337addf"). InnerVolumeSpecName "kube-api-access-mf4xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:29.508574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.508497    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\") pod \"5f94dfa1-639f-4e4d-8104-64059edec661\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") "
Nov 03 21:56:29.508574 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.508566    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\" (UniqueName: \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\") pod \"5f94dfa1-639f-4e4d-8104-64059edec661\" (UID: \"5f94dfa1-639f-4e4d-8104-64059edec661\") "
Nov 03 21:56:29.508912 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.508892    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf4xj\" (UniqueName: \"kubernetes.io/projected/f930e11a-c9d6-40b4-b89f-c266e337addf-kube-api-access-mf4xj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.509013 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.508979    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume" pod "5f94dfa1-639f-4e4d-8104-64059edec661" (UID: "5f94dfa1-639f-4e4d-8104-64059edec661"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:29.510802 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.510772    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb" pod "5f94dfa1-639f-4e4d-8104-64059edec661" (UID: "5f94dfa1-639f-4e4d-8104-64059edec661"). InnerVolumeSpecName "kube-api-access-tr6vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:29.609645 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.609615    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr6vb\" (UniqueName: \"kubernetes.io/projected/5f94dfa1-639f-4e4d-8104-64059edec661-kube-api-access-tr6vb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.609756 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.609655    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94dfa1-639f-4e4d-8104-64059edec661-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.783903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.783835    5533 generic.go:356] "Generic (PLEG): container finished" podID="5f94dfa1-639f-4e4d-8104-64059edec661" containerID="12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40" exitCode=0
Nov 03 21:56:29.783903 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.783879    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"
Nov 03 21:56:29.784115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.783917    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" event={"ID":"5f94dfa1-639f-4e4d-8104-64059edec661","Type":"ContainerDied","Data":"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"}
Nov 03 21:56:29.784115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.783970    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw" event={"ID":"5f94dfa1-639f-4e4d-8104-64059edec661","Type":"ContainerDied","Data":"42edc467f6790759183540cf4018a1aad271f2593a3a402f019cd3c973dc6a3a"}
Nov 03 21:56:29.784115 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.783998    5533 scope.go:122] "RemoveContainer" containerID="12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"
Nov 03 21:56:29.785705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.785681    5533 generic.go:356] "Generic (PLEG): container finished" podID="f930e11a-c9d6-40b4-b89f-c266e337addf" containerID="2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f" exitCode=0
Nov 03 21:56:29.785879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.785751    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"
Nov 03 21:56:29.785879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.785802    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" event={"ID":"f930e11a-c9d6-40b4-b89f-c266e337addf","Type":"ContainerDied","Data":"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"}
Nov 03 21:56:29.785879 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.785854    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd" event={"ID":"f930e11a-c9d6-40b4-b89f-c266e337addf","Type":"ContainerDied","Data":"ec0b18f52cca8478329214f825c209a48876687d9cd99fac2aa18e4f22854887"}
Nov 03 21:56:29.799185 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.799150    5533 scope.go:122] "RemoveContainer" containerID="12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"
Nov 03 21:56:29.799455 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:29.799432    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40\": container with ID starting with 12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40 not found: ID does not exist" containerID="12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"
Nov 03 21:56:29.799543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.799462    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40"} err="failed to get container status \"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40\": rpc error: code = NotFound desc = could not find container \"12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40\": container with ID starting with 12e36b011ef57ef5c72de29faeb77a577804f6559b83765ec6330e1dbab5cd40 not found: ID does not exist"
Nov 03 21:56:29.799543 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.799484    5533 scope.go:122] "RemoveContainer" containerID="2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"
Nov 03 21:56:29.808305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.808288    5533 scope.go:122] "RemoveContainer" containerID="2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"
Nov 03 21:56:29.808565 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:29.808546    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f\": container with ID starting with 2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f not found: ID does not exist" containerID="2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"
Nov 03 21:56:29.808661 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.808576    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f"} err="failed to get container status \"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f\": rpc error: code = NotFound desc = could not find container \"2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f\": container with ID starting with 2968401ce955c5d773ea27f58bac5eb5c8fed064fef5f91029c2cc8ff4836f6f not found: ID does not exist"
Nov 03 21:56:29.938888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.938864    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:56:29.988126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:29.988087    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a0da1-776677f9c9-lsvvw"]
Nov 03 21:56:30.047555 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:30.047434    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:56:30.051028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:30.051007    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a0da1-6c6b68f768-2bsxd"]
Nov 03 21:56:30.493449 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:30.493419    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f94dfa1-639f-4e4d-8104-64059edec661" path="/var/lib/kubelet/pods/5f94dfa1-639f-4e4d-8104-64059edec661/volumes"
Nov 03 21:56:30.493859 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:30.493844    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f930e11a-c9d6-40b4-b89f-c266e337addf" path="/var/lib/kubelet/pods/f930e11a-c9d6-40b4-b89f-c266e337addf/volumes"
Nov 03 21:56:41.946551 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:56:41.946520    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:56:49.259916 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:56:49.259879    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:57:19.263039 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:57:19.262994    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:57:28.471999 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.471957    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"]
Nov 03 21:57:28.472780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472758    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f930e11a-c9d6-40b4-b89f-c266e337addf" containerName="payload-processing"
Nov 03 21:57:28.472780 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472780    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f930e11a-c9d6-40b4-b89f-c266e337addf" containerName="payload-processing"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472804    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472812    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472835    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b3794b91-e9a3-4d98-bf47-a0003d7b0206" containerName="revoke-keys"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472843    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3794b91-e9a3-4d98-bf47-a0003d7b0206" containerName="revoke-keys"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472881    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5f94dfa1-639f-4e4d-8104-64059edec661" containerName="payload-pre-processing"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472889    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f94dfa1-639f-4e4d-8104-64059edec661" containerName="payload-pre-processing"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472913    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:28.472978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.472959    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:28.473399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.473129    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f930e11a-c9d6-40b4-b89f-c266e337addf" containerName="payload-processing"
Nov 03 21:57:28.473399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.473147    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:28.473399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.473160    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3794b91-e9a3-4d98-bf47-a0003d7b0206" containerName="revoke-keys"
Nov 03 21:57:28.473399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.473176    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f94dfa1-639f-4e4d-8104-64059edec661" containerName="payload-pre-processing"
Nov 03 21:57:28.473399 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.473187    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2aed43-c8de-4517-9b6b-051f83195ef2" containerName="istio-proxy"
Nov 03 21:57:28.476613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.476592    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.527330 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.527308    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:57:28.573951 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.573923    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 21:57:28.577146 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.577125    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"]
Nov 03 21:57:28.615762 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.615728    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.615885 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.615866    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.615982 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.615965    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zmw\" (UniqueName: \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.717291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.717261    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.717407 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.717351    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.717455 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.717440    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zmw\" (UniqueName: \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.717996 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.717974    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.719720 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.719699    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.775603 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.775524    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zmw\" (UniqueName: \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\") pod \"maas-api-revoke-sub-ee1d62bf-277mz\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:28.790599 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:28.790566    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:29.160289 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:29.160254    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"]
Nov 03 21:57:29.162297 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:57:29.162270    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdf96d2_67ff_43d3_9faf_b05bf08a2a57.slice/crio-9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20 WatchSource:0}: Error finding container 9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20: Status 404 returned error can't find the container with id 9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20
Nov 03 21:57:29.200325 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:29.200285    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz" event={"ID":"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57","Type":"ContainerStarted","Data":"9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20"}
Nov 03 21:57:30.209863 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:30.209827    5533 generic.go:356] "Generic (PLEG): container finished" podID="4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" containerID="dcc9bec5cbac01e09424a7ccec82b9eafbda9003733fe4e4a661c27e146fbf2b" exitCode=0
Nov 03 21:57:30.210305 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:30.209927    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz" event={"ID":"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57","Type":"ContainerDied","Data":"dcc9bec5cbac01e09424a7ccec82b9eafbda9003733fe4e4a661c27e146fbf2b"}
Nov 03 21:57:31.424405 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.424381    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:31.548828 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.548761    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\" (UniqueName: \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\") pod \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") "
Nov 03 21:57:31.548975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.548829    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\") pod \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") "
Nov 03 21:57:31.548975 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.548901    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\") pod \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\" (UID: \"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57\") "
Nov 03 21:57:31.549433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.549317    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca" pod "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" (UID: "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:31.549433 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.549411    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:57:31.551126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.551101    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca" pod "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" (UID: "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:31.551126 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.551111    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw" pod "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" (UID: "4cdf96d2-67ff-43d3-9faf-b05bf08a2a57"). InnerVolumeSpecName "kube-api-access-n5zmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:31.650066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.650040    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zmw\" (UniqueName: \"kubernetes.io/projected/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-kube-api-access-n5zmw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:57:31.650177 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:31.650067    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:57:32.225203 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:32.225169    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"
Nov 03 21:57:32.225376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:32.225178    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz" event={"ID":"4cdf96d2-67ff-43d3-9faf-b05bf08a2a57","Type":"ContainerDied","Data":"9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20"}
Nov 03 21:57:32.225376 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:32.225292    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9559b911e98a78d4ddb84a10ec1c8a867e2b8e1c7d3fc7e974e1ebd445d0ac20"
Nov 03 21:57:35.629888 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.629849    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 21:57:35.630705 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.630683    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" containerName="revoke-keys"
Nov 03 21:57:35.630846 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.630706    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" containerName="revoke-keys"
Nov 03 21:57:35.630917 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.630901    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" containerName="revoke-keys"
Nov 03 21:57:35.630978 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.630926    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd96c5b1-1787-4cc6-b38e-d9eec976a039" containerName="istio-proxy"
Nov 03 21:57:35.634577 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.634553    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.732810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.732766    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"kube-root-ca.crt\""
Nov 03 21:57:35.732810 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.732807    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"openshift-service-ca.crt\""
Nov 03 21:57:35.733002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.732826    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"default-dockercfg-z6nqg\""
Nov 03 21:57:35.733002 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.732883    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c11538\"/\"sub-test-model-c11538-kserve-self-signed-certs\""
Nov 03 21:57:35.735997 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.735977    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 21:57:35.795526 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.795483    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.795897 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.795876    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.796296 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.796059    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.796567 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.796542    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.797246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.797215    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpss\" (UniqueName: \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.797402 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.797281    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.897908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.897840    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.897908 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.897881    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898081 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.897947    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpss\" (UniqueName: \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898160 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898082    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898162    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898222 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898194    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898344 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898308    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898456 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898432    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.898508 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.898491    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.900150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.900127    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.900486 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.900469    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:35.956541 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:35.956519    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpss\" (UniqueName: \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\") pod \"sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") " pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:36.248492 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:36.248460    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:36.627323 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:36.627290    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 21:57:36.628259 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:57:36.628236    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9488aa2b_0f03_492a_af09_fc4c8b704b89.slice/crio-1b4ee237a059a3e03b246ac2209a4ad673e0c8c0b15ca69570f13944971b0963 WatchSource:0}: Error finding container 1b4ee237a059a3e03b246ac2209a4ad673e0c8c0b15ca69570f13944971b0963: Status 404 returned error can't find the container with id 1b4ee237a059a3e03b246ac2209a4ad673e0c8c0b15ca69570f13944971b0963
Nov 03 21:57:37.261418 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:37.261373    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" event={"ID":"9488aa2b-0f03-492a-af09-fc4c8b704b89","Type":"ContainerStarted","Data":"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"}
Nov 03 21:57:37.261853 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:37.261434    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" event={"ID":"9488aa2b-0f03-492a-af09-fc4c8b704b89","Type":"ContainerStarted","Data":"1b4ee237a059a3e03b246ac2209a4ad673e0c8c0b15ca69570f13944971b0963"}
Nov 03 21:57:37.372817 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:37.372770    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" podStartSLOduration=2.372756682 podStartE2EDuration="2.372756682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:37.371163353 +0000 UTC m=+4159.002476829" watchObservedRunningTime="2026-11-03 21:57:37.372756682 +0000 UTC m=+4159.004070158"
Nov 03 21:57:38.248784 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:38.248755    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:42.739738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:42.739705    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:57:46.249401 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:46.249369    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:46.261833 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:46.261808    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:46.337954 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:57:46.337931    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 21:57:49.261870 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:57:49.261829    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:58:19.266306 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:19.266279    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:58:19.268802 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:58:19.266767    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:58:19.277254 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:19.277234    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:58:19.332750 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:19.332729    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 21:58:19.343717 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:19.343699    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 21:58:35.904769 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:35.904733    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:58:43.609028 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.608991    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 21:58:43.613451 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.613430    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.713119 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.713087    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"kube-root-ca.crt\""
Nov 03 21:58:43.713408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.713379    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"default-dockercfg-fbt9z\""
Nov 03 21:58:43.713408 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.713393    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"openshift-service-ca.crt\""
Nov 03 21:58:43.713557 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.713428    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0dc66\"/\"sub-test-model-f0dc66-kserve-self-signed-certs\""
Nov 03 21:58:43.716850 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.716823    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 21:58:43.758830 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.758798    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.759102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.759068    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.759246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.759117    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.759246 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.759149    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgb8c\" (UniqueName: \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.759373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.759258    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.759373 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.759310    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.860741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.860651    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.860741 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.860702    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.860955 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.860875    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.860970    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861016 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.861004    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.861033    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgb8c\" (UniqueName: \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861130 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.861038    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861386 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.861331    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.861510 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.861459    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.863059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.863035    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.863488 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.863470    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.918893 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.918869    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgb8c\" (UniqueName: \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\") pod \"sub-test-model-f0dc66-kserve-8fd645555-rgzxn\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") " pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:43.925629 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:43.925607    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:44.298919 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:44.298884    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 21:58:44.300074 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:58:44.300041    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d0819e6_1fac_476c_8c52_07dec75e9b73.slice/crio-202634cdccc529287e7ed9b5c7674e817ae3de78d965d59a7b290052cf263973 WatchSource:0}: Error finding container 202634cdccc529287e7ed9b5c7674e817ae3de78d965d59a7b290052cf263973: Status 404 returned error can't find the container with id 202634cdccc529287e7ed9b5c7674e817ae3de78d965d59a7b290052cf263973
Nov 03 21:58:44.837988 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:44.837952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" event={"ID":"6d0819e6-1fac-476c-8c52-07dec75e9b73","Type":"ContainerStarted","Data":"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"}
Nov 03 21:58:44.838396 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:44.837993    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" event={"ID":"6d0819e6-1fac-476c-8c52-07dec75e9b73","Type":"ContainerStarted","Data":"202634cdccc529287e7ed9b5c7674e817ae3de78d965d59a7b290052cf263973"}
Nov 03 21:58:44.926274 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:44.926248    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:44.949565 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:44.949512    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" podStartSLOduration=1.9494987830000001 podStartE2EDuration="1.949498783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:44.948377671 +0000 UTC m=+4226.579691151" watchObservedRunningTime="2026-11-03 21:58:44.949498783 +0000 UTC m=+4226.580812260"
Nov 03 21:58:49.261168 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:58:49.261131    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:58:53.926568 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:53.926536    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:53.939070 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:53.939047    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:58:54.924457 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:58:54.924429    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 21:59:11.370059 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.369970    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 21:59:11.374546 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.374525    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.423551 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.423519    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85db4aad-openshift-default-dockercfg-8jgdx\""
Nov 03 21:59:11.475163 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.475124    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 21:59:11.475957 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.475934    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.475983    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476086    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476114    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476150 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476135    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476153    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476219    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476241    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.476379 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.476255    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578062    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578109 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578110    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578189    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578383 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578265    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578544 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578516    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578610 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578562    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578620    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578738 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578709    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578749    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578786    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.578858 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578814    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.579036 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.578949    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.579170 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.579149    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.579902 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.579873    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.580749 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.580731    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.581215 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.581189    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.636709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.636649    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.636709 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.636657    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\") pod \"e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") " pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:11.690074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:11.690052    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:12.122655 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:12.122612    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 21:59:12.123557 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:59:12.123533    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2343faa_fc11_46e1_8251_9339ae36b00a.slice/crio-6808e14126b5d4bb97cc62d5de3ac86b125dbb788b89c22d085bff6933daf789 WatchSource:0}: Error finding container 6808e14126b5d4bb97cc62d5de3ac86b125dbb788b89c22d085bff6933daf789: Status 404 returned error can't find the container with id 6808e14126b5d4bb97cc62d5de3ac86b125dbb788b89c22d085bff6933daf789
Nov 03 21:59:12.126579 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:12.126551    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:59:12.126663 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:12.126610    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:59:12.126711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:12.126673    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:59:13.039047 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:13.039010    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerStarted","Data":"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"}
Nov 03 21:59:13.039500 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:13.039065    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerStarted","Data":"6808e14126b5d4bb97cc62d5de3ac86b125dbb788b89c22d085bff6933daf789"}
Nov 03 21:59:13.201284 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:13.201224    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podStartSLOduration=2.2012084 podStartE2EDuration="2.2012084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:13.199041978 +0000 UTC m=+4254.830355457" watchObservedRunningTime="2026-11-03 21:59:13.2012084 +0000 UTC m=+4254.832521878"
Nov 03 21:59:13.691074 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:13.691044    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:13.697351 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:13.697324    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:14.045316 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:14.045239    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:14.046096 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:14.046079    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 21:59:19.260815 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:59:19.260777    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 21:59:19.802102 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.802065    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 21:59:19.807613 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.807591    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:19.862006 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.861974    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 21:59:19.919711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.901797    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-85db4aad\""
Nov 03 21:59:19.919711 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.901942    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-85db4aad-dockercfg-8mrfm\""
Nov 03 21:59:19.947046 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.943320    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 21:59:19.948263 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.948235    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:19.982755 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.982718    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqdx\" (UniqueName: \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:19.983066 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.983043    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:19.983226 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.983211    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:19.983382 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:19.983366    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqzr\" (UniqueName: \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.005194 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.005164    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 21:59:20.084620 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.084528    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqdx\" (UniqueName: \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:20.084826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.084620    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.084826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.084672    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:20.084826 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.084707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqzr\" (UniqueName: \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.085925 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.085902    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:20.086267 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.086240    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.142869 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.142840    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqzr\" (UniqueName: \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\") pod \"payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") " pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.143116 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.143093    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqdx\" (UniqueName: \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\") pod \"payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:20.293291 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.293263    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:20.424775 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.424741    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:20.673839 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.673809    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 21:59:20.674454 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:59:20.674422    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfb8287_8b18_458a_bb52_aff38468ea2d.slice/crio-b101f2ade9fd602876f87af8ec3db712a86fa0d27253ab1c19b28fc88f6d417a WatchSource:0}: Error finding container b101f2ade9fd602876f87af8ec3db712a86fa0d27253ab1c19b28fc88f6d417a: Status 404 returned error can't find the container with id b101f2ade9fd602876f87af8ec3db712a86fa0d27253ab1c19b28fc88f6d417a
Nov 03 21:59:20.796614 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:20.796587    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 21:59:20.796961 ip-10-0-140-144 kubenswrapper[5533]: W1103 21:59:20.796936    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97549b96_4caf_47b2_86ca_a471022a8af1.slice/crio-ddf11aba2ac962164898e9575479df094b6296089bc1f527eb978a9fb55e5b55 WatchSource:0}: Error finding container ddf11aba2ac962164898e9575479df094b6296089bc1f527eb978a9fb55e5b55: Status 404 returned error can't find the container with id ddf11aba2ac962164898e9575479df094b6296089bc1f527eb978a9fb55e5b55
Nov 03 21:59:21.097336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.097243    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" event={"ID":"5cfb8287-8b18-458a-bb52-aff38468ea2d","Type":"ContainerStarted","Data":"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"}
Nov 03 21:59:21.097336 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.097309    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" event={"ID":"5cfb8287-8b18-458a-bb52-aff38468ea2d","Type":"ContainerStarted","Data":"b101f2ade9fd602876f87af8ec3db712a86fa0d27253ab1c19b28fc88f6d417a"}
Nov 03 21:59:21.097602 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.097579    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:21.099054 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.099028    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" event={"ID":"97549b96-4caf-47b2-86ca-a471022a8af1","Type":"ContainerStarted","Data":"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"}
Nov 03 21:59:21.099190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.099075    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:21.099190 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.099116    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" event={"ID":"97549b96-4caf-47b2-86ca-a471022a8af1","Type":"ContainerStarted","Data":"ddf11aba2ac962164898e9575479df094b6296089bc1f527eb978a9fb55e5b55"}
Nov 03 21:59:21.213358 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.213299    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" podStartSLOduration=2.213281741 podStartE2EDuration="2.213281741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:21.206016738 +0000 UTC m=+4262.837330216" watchObservedRunningTime="2026-11-03 21:59:21.213281741 +0000 UTC m=+4262.844595219"
Nov 03 21:59:21.316345 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:21.316290    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" podStartSLOduration=2.316277327 podStartE2EDuration="2.316277327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:21.314797868 +0000 UTC m=+4262.946111344" watchObservedRunningTime="2026-11-03 21:59:21.316277327 +0000 UTC m=+4262.947590804"
Nov 03 21:59:32.111498 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:32.111467    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 21:59:32.112041 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:32.111550    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 21:59:32.932901 ip-10-0-140-144 kubenswrapper[5533]: I1103 21:59:32.932867    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 21:59:49.262165 ip-10-0-140-144 kubenswrapper[5533]: E1103 21:59:49.262114    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:00:00.185755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.185724    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:00.189910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.189887    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:00.190046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.190017    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.194155 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.194127    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:00.194286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.194251    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.197942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.197923    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.232120 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.232099    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:00.244245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.244219    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:00.244363 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.244335    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.248373 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.248354    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:00.248498 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.248462    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.252239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.252221    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:00.252346 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.252330    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.255786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.255770    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:00.255862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.255850    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.259464 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.259416    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:00.259564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.259544    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.263265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.263245    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.275942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.275921    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:00:00.282705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.282685    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:00.287050 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.287029    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:00.291253 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.291231    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:00.319885 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.319849    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.320104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320077    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.320274 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320253    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456rd\" (UniqueName: \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.320427 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320410    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.320599 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320558    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts58z\" (UniqueName: \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.320821 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320790    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.321021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.320997    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48kx\" (UniqueName: \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.321121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321087    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftfx\" (UniqueName: \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.321188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321144    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5pk\" (UniqueName: \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.321188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321179    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.321308 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321227    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hkg\" (UniqueName: \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.321308 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321267    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.321423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321309    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.321423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321333    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.321423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321382    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696mh\" (UniqueName: \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.321423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321421    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.321670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321447    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.321670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.321522    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.329991 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.329966    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:00.350974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.349131    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:00.352779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.352757    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:00.356937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.356916    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:00.361656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.361617    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:00.389610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.389588    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:00.422502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422467    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696mh\" (UniqueName: \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.422666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422519    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.422666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422560    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.422819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422671    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.422819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422796    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.422953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422830    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.422953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422862    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456rd\" (UniqueName: \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.422953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422892    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.422953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422912    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts58z\" (UniqueName: \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.422953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422929    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.422956    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48kx\" (UniqueName: \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423008    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftfx\" (UniqueName: \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423033    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5pk\" (UniqueName: \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423057    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423077    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hkg\" (UniqueName: \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423110    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423136    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.423272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423152    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.423806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423340    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.423806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423393    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.423806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.423806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423550    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.423806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.423763    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.424245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.424223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.424286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.424223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.424551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.424528    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.424590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.424580    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.481385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.481285    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.481385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.481295    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48kx\" (UniqueName: \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.481385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.481295    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696mh\" (UniqueName: \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\") pod \"maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.481676 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.481657    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftfx\" (UniqueName: \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.485493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.485471    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts58z\" (UniqueName: \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\") pod \"maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.485756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.485728    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.486418 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.486393    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5pk\" (UniqueName: \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.509277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.509257    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:00.512733 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.512712    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456rd\" (UniqueName: \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\") pod \"maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.512972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.512952    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hkg\" (UniqueName: \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.515402 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.515382    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:00.522615 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.522594    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:00.561800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.561066    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:00.573830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.573803    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:00.579657 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.579617    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:00.588554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.588013    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:00.598677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.597371    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:00.608211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.607849    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:00.917962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.917650    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:00.963187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.963106    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:00.968093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:00.967280    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:00.969695 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:00.968726    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4003a9_ed80_4cd6_a12a_c22a5ada9600.slice/crio-084074f64499404ac5bc84df3061b67e3901aa6768d0f3f937846799cfcdea67 WatchSource:0}: Error finding container 084074f64499404ac5bc84df3061b67e3901aa6768d0f3f937846799cfcdea67: Status 404 returned error can't find the container with id 084074f64499404ac5bc84df3061b67e3901aa6768d0f3f937846799cfcdea67
Nov 03 22:00:00.969819 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:00.969742    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5153f1a0_7a0e_44b4_acc9_578f874f06cd.slice/crio-112048949c32679bfb3f95e75913ea696d4268202c63447617953732ba0a8a7c WatchSource:0}: Error finding container 112048949c32679bfb3f95e75913ea696d4268202c63447617953732ba0a8a7c: Status 404 returned error can't find the container with id 112048949c32679bfb3f95e75913ea696d4268202c63447617953732ba0a8a7c
Nov 03 22:00:01.073914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.073883    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:01.074472 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:01.074448    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c96786_1d79_42f7_b5dd_2a666768d64c.slice/crio-44dbafe245d379edc937de652be4649835b2f0c27492e57ef28f47417a691412 WatchSource:0}: Error finding container 44dbafe245d379edc937de652be4649835b2f0c27492e57ef28f47417a691412: Status 404 returned error can't find the container with id 44dbafe245d379edc937de652be4649835b2f0c27492e57ef28f47417a691412
Nov 03 22:00:01.078298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.078266    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:01.089361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.089337    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:01.107223 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:01.107192    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc193516f_b5cc_457c_a205_d9a869bd4241.slice/crio-19325db7b3e6ee0d9284a52d981c1f08107c473b1de4b981eb8268a9531da5f4 WatchSource:0}: Error finding container 19325db7b3e6ee0d9284a52d981c1f08107c473b1de4b981eb8268a9531da5f4: Status 404 returned error can't find the container with id 19325db7b3e6ee0d9284a52d981c1f08107c473b1de4b981eb8268a9531da5f4
Nov 03 22:00:01.107727 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.107703    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:01.107862 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:01.107799    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ff8a62_56b4_47af_be01_09eb41036b4d.slice/crio-c79bae680baccfbfaaa5b1380c7cd33f682bd74c16e7ad2f42b35f43e478fc45 WatchSource:0}: Error finding container c79bae680baccfbfaaa5b1380c7cd33f682bd74c16e7ad2f42b35f43e478fc45: Status 404 returned error can't find the container with id c79bae680baccfbfaaa5b1380c7cd33f682bd74c16e7ad2f42b35f43e478fc45
Nov 03 22:00:01.117542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.117516    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:01.122721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.122699    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:01.154071 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:01.154032    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc488d1aa_d8a3_4a7d_b041_1ad5acbd7f31.slice/crio-eb39e015e7c0df7c3b51b7e0ae017690127615d6c16ec2a9c9d9fecc81df9e10 WatchSource:0}: Error finding container eb39e015e7c0df7c3b51b7e0ae017690127615d6c16ec2a9c9d9fecc81df9e10: Status 404 returned error can't find the container with id eb39e015e7c0df7c3b51b7e0ae017690127615d6c16ec2a9c9d9fecc81df9e10
Nov 03 22:00:01.155182 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:00:01.155157    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6175aca6_6a81_4be9_9e6e_9678fe482481.slice/crio-ee785e4d9f41e978bfa3cd88d7f42f825d2db69b147ee6d6694d63456c91d838 WatchSource:0}: Error finding container ee785e4d9f41e978bfa3cd88d7f42f825d2db69b147ee6d6694d63456c91d838: Status 404 returned error can't find the container with id ee785e4d9f41e978bfa3cd88d7f42f825d2db69b147ee6d6694d63456c91d838
Nov 03 22:00:01.394820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.394787    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerStarted","Data":"eb39e015e7c0df7c3b51b7e0ae017690127615d6c16ec2a9c9d9fecc81df9e10"}
Nov 03 22:00:01.396395 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.396369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerStarted","Data":"c79bae680baccfbfaaa5b1380c7cd33f682bd74c16e7ad2f42b35f43e478fc45"}
Nov 03 22:00:01.398862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.398269    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerStarted","Data":"57da1ecd7a273c5ac28069e6d0190e001a46d9867582e620219ed623f8ae55e9"}
Nov 03 22:00:01.398862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.398320    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerStarted","Data":"084074f64499404ac5bc84df3061b67e3901aa6768d0f3f937846799cfcdea67"}
Nov 03 22:00:01.400105 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.400080    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerStarted","Data":"6b0f252c1cb93cc72246fc37bbb2de035e8b3d2659aac139db865664080ce8ce"}
Nov 03 22:00:01.400197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.400128    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerStarted","Data":"112048949c32679bfb3f95e75913ea696d4268202c63447617953732ba0a8a7c"}
Nov 03 22:00:01.401515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.401493    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerStarted","Data":"ee785e4d9f41e978bfa3cd88d7f42f825d2db69b147ee6d6694d63456c91d838"}
Nov 03 22:00:01.402919 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.402898    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerStarted","Data":"009db869750fcc6a60d1d1bd0c220b58d802873199bbad1ff412ff6cfaeb9c8c"}
Nov 03 22:00:01.404284 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.404258    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerStarted","Data":"028f45f15039d6cad27bb6db8e615fc0234435bcd9e47459fe15d81c8de0b464"}
Nov 03 22:00:01.405545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.405525    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerStarted","Data":"19325db7b3e6ee0d9284a52d981c1f08107c473b1de4b981eb8268a9531da5f4"}
Nov 03 22:00:01.406910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.406883    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerStarted","Data":"44dbafe245d379edc937de652be4649835b2f0c27492e57ef28f47417a691412"}
Nov 03 22:00:01.510361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.510287    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" podStartSLOduration=1.5102750839999999 podStartE2EDuration="1.510275084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.506364527 +0000 UTC m=+4303.137678003" watchObservedRunningTime="2026-11-03 22:00:01.510275084 +0000 UTC m=+4303.141588561"
Nov 03 22:00:01.618671 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:01.618606    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" podStartSLOduration=1.618592743 podStartE2EDuration="1.618592743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.61576876 +0000 UTC m=+4303.247082242" watchObservedRunningTime="2026-11-03 22:00:01.618592743 +0000 UTC m=+4303.249906232"
Nov 03 22:00:02.415569 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.415532    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerStarted","Data":"0b5b79a7dc901bed16a56c1cda1bc4ab884202e85b3d8bac9411c5a8767975fa"}
Nov 03 22:00:02.417147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.417114    5533 generic.go:356] "Generic (PLEG): container finished" podID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerID="b7dcf5fc64abb010ca744a62cae68db9efe3462d418029dfecc8eeb447fae801" exitCode=60
Nov 03 22:00:02.417257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.417163    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerDied","Data":"b7dcf5fc64abb010ca744a62cae68db9efe3462d418029dfecc8eeb447fae801"}
Nov 03 22:00:02.417499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.417481    5533 scope.go:122] "RemoveContainer" containerID="b7dcf5fc64abb010ca744a62cae68db9efe3462d418029dfecc8eeb447fae801"
Nov 03 22:00:02.419274 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.419253    5533 generic.go:356] "Generic (PLEG): container finished" podID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerID="57da1ecd7a273c5ac28069e6d0190e001a46d9867582e620219ed623f8ae55e9" exitCode=60
Nov 03 22:00:02.419381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.419328    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerDied","Data":"57da1ecd7a273c5ac28069e6d0190e001a46d9867582e620219ed623f8ae55e9"}
Nov 03 22:00:02.419515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.419499    5533 scope.go:122] "RemoveContainer" containerID="57da1ecd7a273c5ac28069e6d0190e001a46d9867582e620219ed623f8ae55e9"
Nov 03 22:00:02.421519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.421501    5533 generic.go:356] "Generic (PLEG): container finished" podID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerID="6b0f252c1cb93cc72246fc37bbb2de035e8b3d2659aac139db865664080ce8ce" exitCode=60
Nov 03 22:00:02.421671 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.421625    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerDied","Data":"6b0f252c1cb93cc72246fc37bbb2de035e8b3d2659aac139db865664080ce8ce"}
Nov 03 22:00:02.421952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.421935    5533 scope.go:122] "RemoveContainer" containerID="6b0f252c1cb93cc72246fc37bbb2de035e8b3d2659aac139db865664080ce8ce"
Nov 03 22:00:02.423779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.423754    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerStarted","Data":"1512a5464d80a4e3baa6de5ba10a7653b032baedf5bc3c24d8a9272aa7c859d4"}
Nov 03 22:00:02.425310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.425292    5533 generic.go:356] "Generic (PLEG): container finished" podID="955621e8-ca4c-476c-af40-b8f3186943b0" containerID="882ab1bb11274ab75352989f51df53c7f6dce3b5d316c3d25a2dd5fc0bab0b0e" exitCode=60
Nov 03 22:00:02.425425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.425316    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerDied","Data":"882ab1bb11274ab75352989f51df53c7f6dce3b5d316c3d25a2dd5fc0bab0b0e"}
Nov 03 22:00:02.425548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.425533    5533 scope.go:122] "RemoveContainer" containerID="882ab1bb11274ab75352989f51df53c7f6dce3b5d316c3d25a2dd5fc0bab0b0e"
Nov 03 22:00:02.426949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.426928    5533 generic.go:356] "Generic (PLEG): container finished" podID="6ffe7203-bef8-44df-b103-807099ab28f2" containerID="6b7650231f3ff8c25a2057514d4dfeab9a2bedb035b093bb1627a4ac231f8952" exitCode=60
Nov 03 22:00:02.427039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.426991    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerDied","Data":"6b7650231f3ff8c25a2057514d4dfeab9a2bedb035b093bb1627a4ac231f8952"}
Nov 03 22:00:02.427173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.427160    5533 scope.go:122] "RemoveContainer" containerID="6b7650231f3ff8c25a2057514d4dfeab9a2bedb035b093bb1627a4ac231f8952"
Nov 03 22:00:02.429331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.429285    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerStarted","Data":"90232e6a0f6f874fdea484c6d36bb18a2c8460cec547ad03f99ec3b070b29533"}
Nov 03 22:00:02.431101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.431071    5533 generic.go:356] "Generic (PLEG): container finished" podID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerID="03a09578dfde87f3ea51c04d00390b5eb419e5458f4153308d01110e7a9b11bc" exitCode=60
Nov 03 22:00:02.431211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.431114    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerDied","Data":"03a09578dfde87f3ea51c04d00390b5eb419e5458f4153308d01110e7a9b11bc"}
Nov 03 22:00:02.431380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.431364    5533 scope.go:122] "RemoveContainer" containerID="03a09578dfde87f3ea51c04d00390b5eb419e5458f4153308d01110e7a9b11bc"
Nov 03 22:00:02.526773 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.526724    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" podStartSLOduration=2.526700419 podStartE2EDuration="2.526700419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.523708651 +0000 UTC m=+4304.155022129" watchObservedRunningTime="2026-11-03 22:00:02.526700419 +0000 UTC m=+4304.158013896"
Nov 03 22:00:02.971853 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:02.967099    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" podStartSLOduration=2.967078774 podStartE2EDuration="2.967078774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:02.963992915 +0000 UTC m=+4304.595306393" watchObservedRunningTime="2026-11-03 22:00:02.967078774 +0000 UTC m=+4304.598392259"
Nov 03 22:00:03.218080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.218034    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" podStartSLOduration=3.218019435 podStartE2EDuration="3.218019435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:03.214985191 +0000 UTC m=+4304.846298668" watchObservedRunningTime="2026-11-03 22:00:03.218019435 +0000 UTC m=+4304.849332911"
Nov 03 22:00:03.440209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.440178    5533 generic.go:356] "Generic (PLEG): container finished" podID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerID="1512a5464d80a4e3baa6de5ba10a7653b032baedf5bc3c24d8a9272aa7c859d4" exitCode=60
Nov 03 22:00:03.440624 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.440217    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerDied","Data":"1512a5464d80a4e3baa6de5ba10a7653b032baedf5bc3c24d8a9272aa7c859d4"}
Nov 03 22:00:03.440624 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.440491    5533 scope.go:122] "RemoveContainer" containerID="1512a5464d80a4e3baa6de5ba10a7653b032baedf5bc3c24d8a9272aa7c859d4"
Nov 03 22:00:03.442407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.442385    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerStarted","Data":"2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"}
Nov 03 22:00:03.444528 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.444503    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerStarted","Data":"8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"}
Nov 03 22:00:03.460073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.460048    5533 generic.go:356] "Generic (PLEG): container finished" podID="c193516f-b5cc-457c-a205-d9a869bd4241" containerID="90232e6a0f6f874fdea484c6d36bb18a2c8460cec547ad03f99ec3b070b29533" exitCode=60
Nov 03 22:00:03.460211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.460193    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerDied","Data":"90232e6a0f6f874fdea484c6d36bb18a2c8460cec547ad03f99ec3b070b29533"}
Nov 03 22:00:03.460377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.460358    5533 scope.go:122] "RemoveContainer" containerID="90232e6a0f6f874fdea484c6d36bb18a2c8460cec547ad03f99ec3b070b29533"
Nov 03 22:00:03.465669 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.463936    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerStarted","Data":"dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"}
Nov 03 22:00:03.474948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.473411    5533 generic.go:356] "Generic (PLEG): container finished" podID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerID="0b5b79a7dc901bed16a56c1cda1bc4ab884202e85b3d8bac9411c5a8767975fa" exitCode=60
Nov 03 22:00:03.474948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.473488    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerDied","Data":"0b5b79a7dc901bed16a56c1cda1bc4ab884202e85b3d8bac9411c5a8767975fa"}
Nov 03 22:00:03.474948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.473723    5533 scope.go:122] "RemoveContainer" containerID="0b5b79a7dc901bed16a56c1cda1bc4ab884202e85b3d8bac9411c5a8767975fa"
Nov 03 22:00:03.476972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.476924    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerStarted","Data":"76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"}
Nov 03 22:00:03.485968 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.477115    5533 scope.go:122] "RemoveContainer" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"
Nov 03 22:00:03.485968 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:03.477549    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp_odh-ai-gateway-infra(26ff8a62-56b4-47af-be01-09eb41036b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d"
Nov 03 22:00:03.509682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.504690    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerStarted","Data":"38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"}
Nov 03 22:00:03.509682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.507303    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerStarted","Data":"0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"}
Nov 03 22:00:03.660190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.660115    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" podStartSLOduration=3.66009716 podStartE2EDuration="3.66009716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:03.65854112 +0000 UTC m=+4305.289854597" watchObservedRunningTime="2026-11-03 22:00:03.66009716 +0000 UTC m=+4305.291410638"
Nov 03 22:00:03.876627 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:03.876572    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" podStartSLOduration=3.87655808 podStartE2EDuration="3.87655808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:03.875001474 +0000 UTC m=+4305.506314952" watchObservedRunningTime="2026-11-03 22:00:03.87655808 +0000 UTC m=+4305.507871557"
Nov 03 22:00:04.206278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.206224    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" podStartSLOduration=4.206211616 podStartE2EDuration="4.206211616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:04.202569252 +0000 UTC m=+4305.833882740" watchObservedRunningTime="2026-11-03 22:00:04.206211616 +0000 UTC m=+4305.837525092"
Nov 03 22:00:04.515812 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.515776    5533 generic.go:356] "Generic (PLEG): container finished" podID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerID="dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0" exitCode=60
Nov 03 22:00:04.516268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.515842    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerDied","Data":"dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"}
Nov 03 22:00:04.516268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.515903    5533 scope.go:122] "RemoveContainer" containerID="03a09578dfde87f3ea51c04d00390b5eb419e5458f4153308d01110e7a9b11bc"
Nov 03 22:00:04.516268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.516181    5533 scope.go:122] "RemoveContainer" containerID="dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"
Nov 03 22:00:04.516527 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.516499    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh_odh-ai-gateway-infra(24c96786-1d79-42f7-b5dd-2a666768d64c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c"
Nov 03 22:00:04.518001 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.517977    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerStarted","Data":"d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"}
Nov 03 22:00:04.519769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.519750    5533 generic.go:356] "Generic (PLEG): container finished" podID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc" exitCode=60
Nov 03 22:00:04.519881 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.519814    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerDied","Data":"76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"}
Nov 03 22:00:04.520174 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.520154    5533 scope.go:122] "RemoveContainer" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"
Nov 03 22:00:04.520460 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.520435    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp_odh-ai-gateway-infra(26ff8a62-56b4-47af-be01-09eb41036b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d"
Nov 03 22:00:04.521699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.521680    5533 generic.go:356] "Generic (PLEG): container finished" podID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerID="38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5" exitCode=60
Nov 03 22:00:04.521787 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.521760    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerDied","Data":"38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"}
Nov 03 22:00:04.521953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.521936    5533 scope.go:122] "RemoveContainer" containerID="38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"
Nov 03 22:00:04.522177 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.522153    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l_odh-ai-gateway-infra(cf4003a9-ed80-4cd6-a12a-c22a5ada9600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600"
Nov 03 22:00:04.523602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.523581    5533 generic.go:356] "Generic (PLEG): container finished" podID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerID="0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96" exitCode=60
Nov 03 22:00:04.523712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.523678    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerDied","Data":"0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"}
Nov 03 22:00:04.523868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.523855    5533 scope.go:122] "RemoveContainer" containerID="0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"
Nov 03 22:00:04.524093 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.524070    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb_odh-ai-gateway-infra(5153f1a0-7a0e-44b4-acc9-578f874f06cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd"
Nov 03 22:00:04.525609 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.525588    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerStarted","Data":"c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"}
Nov 03 22:00:04.527484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.527462    5533 generic.go:356] "Generic (PLEG): container finished" podID="955621e8-ca4c-476c-af40-b8f3186943b0" containerID="2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6" exitCode=60
Nov 03 22:00:04.527570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.527538    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerDied","Data":"2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"}
Nov 03 22:00:04.527757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.527741    5533 scope.go:122] "RemoveContainer" containerID="2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"
Nov 03 22:00:04.527925 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.527907    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf_odh-ai-gateway-infra(955621e8-ca4c-476c-af40-b8f3186943b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" podUID="955621e8-ca4c-476c-af40-b8f3186943b0"
Nov 03 22:00:04.529244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.529222    5533 generic.go:356] "Generic (PLEG): container finished" podID="6ffe7203-bef8-44df-b103-807099ab28f2" containerID="8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966" exitCode=60
Nov 03 22:00:04.529323 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.529303    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerDied","Data":"8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"}
Nov 03 22:00:04.529457 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.529441    5533 scope.go:122] "RemoveContainer" containerID="8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"
Nov 03 22:00:04.529625 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:04.529608    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv_odh-ai-gateway-infra(6ffe7203-bef8-44df-b103-807099ab28f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" podUID="6ffe7203-bef8-44df-b103-807099ab28f2"
Nov 03 22:00:04.530060 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.530047    5533 scope.go:122] "RemoveContainer" containerID="b7dcf5fc64abb010ca744a62cae68db9efe3462d418029dfecc8eeb447fae801"
Nov 03 22:00:04.531080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.531057    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerStarted","Data":"fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"}
Nov 03 22:00:04.542860 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.542839    5533 scope.go:122] "RemoveContainer" containerID="57da1ecd7a273c5ac28069e6d0190e001a46d9867582e620219ed623f8ae55e9"
Nov 03 22:00:04.560277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.560246    5533 scope.go:122] "RemoveContainer" containerID="6b0f252c1cb93cc72246fc37bbb2de035e8b3d2659aac139db865664080ce8ce"
Nov 03 22:00:04.573121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.573105    5533 scope.go:122] "RemoveContainer" containerID="882ab1bb11274ab75352989f51df53c7f6dce3b5d316c3d25a2dd5fc0bab0b0e"
Nov 03 22:00:04.586487 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:04.586464    5533 scope.go:122] "RemoveContainer" containerID="6b7650231f3ff8c25a2057514d4dfeab9a2bedb035b093bb1627a4ac231f8952"
Nov 03 22:00:05.541453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.541420    5533 scope.go:122] "RemoveContainer" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"
Nov 03 22:00:05.541962 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.541763    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp_odh-ai-gateway-infra(26ff8a62-56b4-47af-be01-09eb41036b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d"
Nov 03 22:00:05.543383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.543360    5533 scope.go:122] "RemoveContainer" containerID="38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"
Nov 03 22:00:05.543695 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.543670    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l_odh-ai-gateway-infra(cf4003a9-ed80-4cd6-a12a-c22a5ada9600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600"
Nov 03 22:00:05.545172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.545148    5533 scope.go:122] "RemoveContainer" containerID="0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"
Nov 03 22:00:05.545425 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.545403    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb_odh-ai-gateway-infra(5153f1a0-7a0e-44b4-acc9-578f874f06cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd"
Nov 03 22:00:05.546854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.546830    5533 generic.go:356] "Generic (PLEG): container finished" podID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerID="c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711" exitCode=60
Nov 03 22:00:05.546958 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.546912    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerDied","Data":"c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"}
Nov 03 22:00:05.547104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.546976    5533 scope.go:122] "RemoveContainer" containerID="1512a5464d80a4e3baa6de5ba10a7653b032baedf5bc3c24d8a9272aa7c859d4"
Nov 03 22:00:05.547210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.547192    5533 scope.go:122] "RemoveContainer" containerID="c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"
Nov 03 22:00:05.547469 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.547450    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv_odh-ai-gateway-infra(6175aca6-6a81-4be9-9e6e-9678fe482481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481"
Nov 03 22:00:05.549245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.549228    5533 scope.go:122] "RemoveContainer" containerID="2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"
Nov 03 22:00:05.549441 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.549417    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf_odh-ai-gateway-infra(955621e8-ca4c-476c-af40-b8f3186943b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" podUID="955621e8-ca4c-476c-af40-b8f3186943b0"
Nov 03 22:00:05.550803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.550786    5533 scope.go:122] "RemoveContainer" containerID="8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"
Nov 03 22:00:05.551033 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.550990    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv_odh-ai-gateway-infra(6ffe7203-bef8-44df-b103-807099ab28f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" podUID="6ffe7203-bef8-44df-b103-807099ab28f2"
Nov 03 22:00:05.552934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.552912    5533 generic.go:356] "Generic (PLEG): container finished" podID="c193516f-b5cc-457c-a205-d9a869bd4241" containerID="fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716" exitCode=60
Nov 03 22:00:05.553050 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.552990    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerDied","Data":"fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"}
Nov 03 22:00:05.553189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.553172    5533 scope.go:122] "RemoveContainer" containerID="fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"
Nov 03 22:00:05.553452 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.553427    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg_odh-ai-gateway-infra(c193516f-b5cc-457c-a205-d9a869bd4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" podUID="c193516f-b5cc-457c-a205-d9a869bd4241"
Nov 03 22:00:05.555372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.555355    5533 scope.go:122] "RemoveContainer" containerID="dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"
Nov 03 22:00:05.555586 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.555565    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh_odh-ai-gateway-infra(24c96786-1d79-42f7-b5dd-2a666768d64c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c"
Nov 03 22:00:05.557486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.557448    5533 generic.go:356] "Generic (PLEG): container finished" podID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerID="d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d" exitCode=60
Nov 03 22:00:05.557486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.557475    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerDied","Data":"d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"}
Nov 03 22:00:05.557902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.557885    5533 scope.go:122] "RemoveContainer" containerID="d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"
Nov 03 22:00:05.558123 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:05.558104    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h_odh-ai-gateway-infra(c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31"
Nov 03 22:00:05.565177 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.565149    5533 scope.go:122] "RemoveContainer" containerID="90232e6a0f6f874fdea484c6d36bb18a2c8460cec547ad03f99ec3b070b29533"
Nov 03 22:00:05.580439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:05.580410    5533 scope.go:122] "RemoveContainer" containerID="0b5b79a7dc901bed16a56c1cda1bc4ab884202e85b3d8bac9411c5a8767975fa"
Nov 03 22:00:06.569952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:06.569922    5533 scope.go:122] "RemoveContainer" containerID="c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"
Nov 03 22:00:06.570513 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:06.570215    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv_odh-ai-gateway-infra(6175aca6-6a81-4be9-9e6e-9678fe482481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481"
Nov 03 22:00:06.571761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:06.571735    5533 scope.go:122] "RemoveContainer" containerID="fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"
Nov 03 22:00:06.572026 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:06.572006    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg_odh-ai-gateway-infra(c193516f-b5cc-457c-a205-d9a869bd4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" podUID="c193516f-b5cc-457c-a205-d9a869bd4241"
Nov 03 22:00:06.573554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:06.573534    5533 scope.go:122] "RemoveContainer" containerID="d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"
Nov 03 22:00:06.573770 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:06.573750    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h_odh-ai-gateway-infra(c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31"
Nov 03 22:00:08.877022 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:08.876982    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:00:15.486413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:15.486383    5533 scope.go:122] "RemoveContainer" containerID="2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"
Nov 03 22:00:16.486841 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:16.486801    5533 scope.go:122] "RemoveContainer" containerID="d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"
Nov 03 22:00:16.685913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:16.685884    5533 generic.go:356] "Generic (PLEG): container finished" podID="955621e8-ca4c-476c-af40-b8f3186943b0" containerID="ee99e25ff7d2a28fec36b466fcb7b21d90a2f5ea2181b935c76cca0c1b543213" exitCode=60
Nov 03 22:00:16.686056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:16.685958    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerDied","Data":"ee99e25ff7d2a28fec36b466fcb7b21d90a2f5ea2181b935c76cca0c1b543213"}
Nov 03 22:00:16.686056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:16.686027    5533 scope.go:122] "RemoveContainer" containerID="2d1fd0e11e73257a2c5a7841ae1a44cfea404b6146c90bc58becf5081a9aa1d6"
Nov 03 22:00:16.686444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:16.686423    5533 scope.go:122] "RemoveContainer" containerID="ee99e25ff7d2a28fec36b466fcb7b21d90a2f5ea2181b935c76cca0c1b543213"
Nov 03 22:00:16.686758 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:16.686720    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf_odh-ai-gateway-infra(955621e8-ca4c-476c-af40-b8f3186943b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" podUID="955621e8-ca4c-476c-af40-b8f3186943b0"
Nov 03 22:00:17.487449 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.487411    5533 scope.go:122] "RemoveContainer" containerID="0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"
Nov 03 22:00:17.487967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.487549    5533 scope.go:122] "RemoveContainer" containerID="c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"
Nov 03 22:00:17.487967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.487685    5533 scope.go:122] "RemoveContainer" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"
Nov 03 22:00:17.487967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.487784    5533 scope.go:122] "RemoveContainer" containerID="38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"
Nov 03 22:00:17.488153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.488034    5533 scope.go:122] "RemoveContainer" containerID="8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"
Nov 03 22:00:17.488241 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.488221    5533 scope.go:122] "RemoveContainer" containerID="dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"
Nov 03 22:00:17.698426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.698371    5533 generic.go:356] "Generic (PLEG): container finished" podID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerID="98e3908b7c0f22e5c9bf6f8bcf9572f109dc839432fcb330e163f40da41f53e7" exitCode=60
Nov 03 22:00:17.698721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.698434    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerDied","Data":"98e3908b7c0f22e5c9bf6f8bcf9572f109dc839432fcb330e163f40da41f53e7"}
Nov 03 22:00:17.698721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.698501    5533 scope.go:122] "RemoveContainer" containerID="d0d759575937b29c0634f834159a8928ffd289893b4f1a65298af25eb60b171d"
Nov 03 22:00:17.699280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.699254    5533 scope.go:122] "RemoveContainer" containerID="98e3908b7c0f22e5c9bf6f8bcf9572f109dc839432fcb330e163f40da41f53e7"
Nov 03 22:00:17.699667 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:17.699612    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h_odh-ai-gateway-infra(c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31"
Nov 03 22:00:17.808926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:17.808857    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:18.080762 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.080738    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:18.247168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.247131    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\" (UniqueName: \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\") pod \"955621e8-ca4c-476c-af40-b8f3186943b0\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") "
Nov 03 22:00:18.247357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.247328    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\") pod \"955621e8-ca4c-476c-af40-b8f3186943b0\" (UID: \"955621e8-ca4c-476c-af40-b8f3186943b0\") "
Nov 03 22:00:18.247694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.247667    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca" pod "955621e8-ca4c-476c-af40-b8f3186943b0" (UID: "955621e8-ca4c-476c-af40-b8f3186943b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:18.249441 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.249416    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx" pod "955621e8-ca4c-476c-af40-b8f3186943b0" (UID: "955621e8-ca4c-476c-af40-b8f3186943b0"). InnerVolumeSpecName "kube-api-access-wftfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:18.348812 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.348751    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955621e8-ca4c-476c-af40-b8f3186943b0-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.348812 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.348774    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftfx\" (UniqueName: \"kubernetes.io/projected/955621e8-ca4c-476c-af40-b8f3186943b0-kube-api-access-wftfx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.714481 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.714454    5533 generic.go:356] "Generic (PLEG): container finished" podID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerID="e61f99e55eb400b73df9c7cc787b7bd7833c00c2d85ca6ed780d18340c6a9e69" exitCode=60
Nov 03 22:00:18.714908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.714531    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerDied","Data":"e61f99e55eb400b73df9c7cc787b7bd7833c00c2d85ca6ed780d18340c6a9e69"}
Nov 03 22:00:18.714908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.714602    5533 scope.go:122] "RemoveContainer" containerID="c0a8cb52b76b31ff4f043aecb71c56bc639d23abe155d410492a5faa16e48711"
Nov 03 22:00:18.715044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.715024    5533 scope.go:122] "RemoveContainer" containerID="e61f99e55eb400b73df9c7cc787b7bd7833c00c2d85ca6ed780d18340c6a9e69"
Nov 03 22:00:18.715327 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:18.715304    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv_odh-ai-gateway-infra(6175aca6-6a81-4be9-9e6e-9678fe482481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481"
Nov 03 22:00:18.717048 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.717027    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf" event={"ID":"955621e8-ca4c-476c-af40-b8f3186943b0","Type":"ContainerDied","Data":"009db869750fcc6a60d1d1bd0c220b58d802873199bbad1ff412ff6cfaeb9c8c"}
Nov 03 22:00:18.717165 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.717103    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"
Nov 03 22:00:18.719744 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.719720    5533 generic.go:356] "Generic (PLEG): container finished" podID="6ffe7203-bef8-44df-b103-807099ab28f2" containerID="1843fb17eddacf63444dd05113e46d66e9690922e71237040c0f1cc4dee4d7c5" exitCode=60
Nov 03 22:00:18.719842 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.719812    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerDied","Data":"1843fb17eddacf63444dd05113e46d66e9690922e71237040c0f1cc4dee4d7c5"}
Nov 03 22:00:18.720199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.720177    5533 scope.go:122] "RemoveContainer" containerID="1843fb17eddacf63444dd05113e46d66e9690922e71237040c0f1cc4dee4d7c5"
Nov 03 22:00:18.720498 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:18.720458    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv_odh-ai-gateway-infra(6ffe7203-bef8-44df-b103-807099ab28f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" podUID="6ffe7203-bef8-44df-b103-807099ab28f2"
Nov 03 22:00:18.722522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.722502    5533 generic.go:356] "Generic (PLEG): container finished" podID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerID="4712d5f80107db98f415f1e9e6b694f61e7308da3310ee48adff2d276f9e39ac" exitCode=60
Nov 03 22:00:18.722655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.722579    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerDied","Data":"4712d5f80107db98f415f1e9e6b694f61e7308da3310ee48adff2d276f9e39ac"}
Nov 03 22:00:18.723040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.722979    5533 scope.go:122] "RemoveContainer" containerID="4712d5f80107db98f415f1e9e6b694f61e7308da3310ee48adff2d276f9e39ac"
Nov 03 22:00:18.723270 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:18.723251    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh_odh-ai-gateway-infra(24c96786-1d79-42f7-b5dd-2a666768d64c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c"
Nov 03 22:00:18.726811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.726793    5533 generic.go:356] "Generic (PLEG): container finished" podID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerID="280068dc8ad99ae9c32df603b7c2d086342fdb10d56814530c337f8b71c734d5" exitCode=60
Nov 03 22:00:18.726893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.726862    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerDied","Data":"280068dc8ad99ae9c32df603b7c2d086342fdb10d56814530c337f8b71c734d5"}
Nov 03 22:00:18.727192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.727172    5533 scope.go:122] "RemoveContainer" containerID="280068dc8ad99ae9c32df603b7c2d086342fdb10d56814530c337f8b71c734d5"
Nov 03 22:00:18.727429 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:18.727408    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp_odh-ai-gateway-infra(26ff8a62-56b4-47af-be01-09eb41036b4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d"
Nov 03 22:00:18.728858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.728831    5533 generic.go:356] "Generic (PLEG): container finished" podID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerID="5f10dc3b30abac8bbf9b0a8f3ebf6cf15feda9b24b370309d33bcbc299aedfb6" exitCode=60
Nov 03 22:00:18.728961 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.728904    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerDied","Data":"5f10dc3b30abac8bbf9b0a8f3ebf6cf15feda9b24b370309d33bcbc299aedfb6"}
Nov 03 22:00:18.729268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.729248    5533 scope.go:122] "RemoveContainer" containerID="5f10dc3b30abac8bbf9b0a8f3ebf6cf15feda9b24b370309d33bcbc299aedfb6"
Nov 03 22:00:18.729504 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:18.729482    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l_odh-ai-gateway-infra(cf4003a9-ed80-4cd6-a12a-c22a5ada9600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600"
Nov 03 22:00:18.730934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.730915    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerStarted","Data":"36c66582d8e5dfcbae27454a901daeb4082e2488905a24af4086e761f30d93c3"}
Nov 03 22:00:18.822979 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.822942    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:18.913933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.913868    5533 scope.go:122] "RemoveContainer" containerID="ee99e25ff7d2a28fec36b466fcb7b21d90a2f5ea2181b935c76cca0c1b543213"
Nov 03 22:00:18.927412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.927384    5533 scope.go:122] "RemoveContainer" containerID="8139b616da11b5c46ff912f632e527f0be63686fe68bf97b988217708b957966"
Nov 03 22:00:18.943354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.943327    5533 scope.go:122] "RemoveContainer" containerID="dd3d5e02a8956e1dbbbead533bdc557a10709ea63eb2cc9968e6b1cf832a5cb0"
Nov 03 22:00:18.958133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.958113    5533 scope.go:122] "RemoveContainer" containerID="76e3fdef95e0683e71e9e8f8872a10f6e79c30e91c0b3c27b1f13dfea40a8efc"
Nov 03 22:00:18.993114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.992829    5533 scope.go:122] "RemoveContainer" containerID="38df778ecce87746f097bd661e4f47cdbaf44712c7a5aa1485abe5ccd9ff63c5"
Nov 03 22:00:18.997694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:18.997670    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:19.160764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.160733    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\") pod \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") "
Nov 03 22:00:19.160938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.160883    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\") pod \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\" (UID: \"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31\") "
Nov 03 22:00:19.161289 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.161233    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca" pod "c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" (UID: "c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:19.163549 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.163525    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h" pod "c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" (UID: "c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31"). InnerVolumeSpecName "kube-api-access-btf2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:19.260698 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:19.260668    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:00:19.262386 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.262352    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.262522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.262387    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btf2h\" (UniqueName: \"kubernetes.io/projected/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31-kube-api-access-btf2h\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.486067 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.486039    5533 scope.go:122] "RemoveContainer" containerID="fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"
Nov 03 22:00:19.648754 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.648722    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:19.653218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.653193    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c11538-29895720-9kvzf"]
Nov 03 22:00:19.739696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.739605    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"
Nov 03 22:00:19.739696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.739630    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h" event={"ID":"c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31","Type":"ContainerDied","Data":"eb39e015e7c0df7c3b51b7e0ae017690127615d6c16ec2a9c9d9fecc81df9e10"}
Nov 03 22:00:19.740089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.739711    5533 scope.go:122] "RemoveContainer" containerID="98e3908b7c0f22e5c9bf6f8bcf9572f109dc839432fcb330e163f40da41f53e7"
Nov 03 22:00:19.745508 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.745485    5533 generic.go:356] "Generic (PLEG): container finished" podID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerID="36c66582d8e5dfcbae27454a901daeb4082e2488905a24af4086e761f30d93c3" exitCode=60
Nov 03 22:00:19.745589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.745560    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerDied","Data":"36c66582d8e5dfcbae27454a901daeb4082e2488905a24af4086e761f30d93c3"}
Nov 03 22:00:19.745999 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.745979    5533 scope.go:122] "RemoveContainer" containerID="36c66582d8e5dfcbae27454a901daeb4082e2488905a24af4086e761f30d93c3"
Nov 03 22:00:19.746299 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:19.746275    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb_odh-ai-gateway-infra(5153f1a0-7a0e-44b4-acc9-578f874f06cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd"
Nov 03 22:00:19.753075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.753060    5533 scope.go:122] "RemoveContainer" containerID="0bd1f185b8caf5d7ee515cec691b7743de054ca1ce73ee9873d42766a0ea0b96"
Nov 03 22:00:19.839702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.839666    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:19.956797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.956758    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:20.015664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:19.996609    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:20.015664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.002579    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a6219b-29895720-8lc5h"]
Nov 03 22:00:20.068300 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.068271    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:20.070112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.070091    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:20.174298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.174265    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\") pod \"6175aca6-6a81-4be9-9e6e-9678fe482481\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") "
Nov 03 22:00:20.174517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.174497    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\") pod \"6175aca6-6a81-4be9-9e6e-9678fe482481\" (UID: \"6175aca6-6a81-4be9-9e6e-9678fe482481\") "
Nov 03 22:00:20.176026 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.175714    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca" pod "6175aca6-6a81-4be9-9e6e-9678fe482481" (UID: "6175aca6-6a81-4be9-9e6e-9678fe482481"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.179870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.179843    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj" pod "6175aca6-6a81-4be9-9e6e-9678fe482481" (UID: "6175aca6-6a81-4be9-9e6e-9678fe482481"). InnerVolumeSpecName "kube-api-access-4gccj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.182859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.182832    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:20.226251 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.226224    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:20.246628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.246578    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:20.276115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.276092    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175aca6-6a81-4be9-9e6e-9678fe482481-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.276115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.276115    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gccj\" (UniqueName: \"kubernetes.io/projected/6175aca6-6a81-4be9-9e6e-9678fe482481-kube-api-access-4gccj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.335394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.335367    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:20.377126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377096    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\") pod \"26ff8a62-56b4-47af-be01-09eb41036b4d\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") "
Nov 03 22:00:20.377288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377204    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\" (UniqueName: \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\") pod \"26ff8a62-56b4-47af-be01-09eb41036b4d\" (UID: \"26ff8a62-56b4-47af-be01-09eb41036b4d\") "
Nov 03 22:00:20.377288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377241    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\" (UniqueName: \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\") pod \"24c96786-1d79-42f7-b5dd-2a666768d64c\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") "
Nov 03 22:00:20.377369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377293    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\") pod \"24c96786-1d79-42f7-b5dd-2a666768d64c\" (UID: \"24c96786-1d79-42f7-b5dd-2a666768d64c\") "
Nov 03 22:00:20.378019 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377982    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca" pod "24c96786-1d79-42f7-b5dd-2a666768d64c" (UID: "24c96786-1d79-42f7-b5dd-2a666768d64c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.378129 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.377991    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca" pod "26ff8a62-56b4-47af-be01-09eb41036b4d" (UID: "26ff8a62-56b4-47af-be01-09eb41036b4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.379854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.379833    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh" pod "26ff8a62-56b4-47af-be01-09eb41036b4d" (UID: "26ff8a62-56b4-47af-be01-09eb41036b4d"). InnerVolumeSpecName "kube-api-access-696mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.380509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.380485    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z" pod "24c96786-1d79-42f7-b5dd-2a666768d64c" (UID: "24c96786-1d79-42f7-b5dd-2a666768d64c"). InnerVolumeSpecName "kube-api-access-ts58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.482667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.480289    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff8a62-56b4-47af-be01-09eb41036b4d-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.482667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.480326    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696mh\" (UniqueName: \"kubernetes.io/projected/26ff8a62-56b4-47af-be01-09eb41036b4d-kube-api-access-696mh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.482667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.480342    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts58z\" (UniqueName: \"kubernetes.io/projected/24c96786-1d79-42f7-b5dd-2a666768d64c-kube-api-access-ts58z\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.482667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.480358    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c96786-1d79-42f7-b5dd-2a666768d64c-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.484780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.484748    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:20.496863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.496806    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" path="/var/lib/kubelet/pods/955621e8-ca4c-476c-af40-b8f3186943b0/volumes"
Nov 03 22:00:20.497430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.497412    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" path="/var/lib/kubelet/pods/c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31/volumes"
Nov 03 22:00:20.556952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.556798    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:20.585175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.585144    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\") pod \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") "
Nov 03 22:00:20.585317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.585262    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\" (UniqueName: \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\") pod \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\" (UID: \"cf4003a9-ed80-4cd6-a12a-c22a5ada9600\") "
Nov 03 22:00:20.587479 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.586588    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca" pod "cf4003a9-ed80-4cd6-a12a-c22a5ada9600" (UID: "cf4003a9-ed80-4cd6-a12a-c22a5ada9600"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.597210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.596806    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk" pod "cf4003a9-ed80-4cd6-a12a-c22a5ada9600" (UID: "cf4003a9-ed80-4cd6-a12a-c22a5ada9600"). InnerVolumeSpecName "kube-api-access-gr5pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.687454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.687420    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.687454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.687455    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5pk\" (UniqueName: \"kubernetes.io/projected/cf4003a9-ed80-4cd6-a12a-c22a5ada9600-kube-api-access-gr5pk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.763051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.762978    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"
Nov 03 22:00:20.763462 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.762982    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp" event={"ID":"26ff8a62-56b4-47af-be01-09eb41036b4d","Type":"ContainerDied","Data":"c79bae680baccfbfaaa5b1380c7cd33f682bd74c16e7ad2f42b35f43e478fc45"}
Nov 03 22:00:20.763462 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.763150    5533 scope.go:122] "RemoveContainer" containerID="280068dc8ad99ae9c32df603b7c2d086342fdb10d56814530c337f8b71c734d5"
Nov 03 22:00:20.763577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.763508    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:20.764973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.764955    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"
Nov 03 22:00:20.765098 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.764988    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l" event={"ID":"cf4003a9-ed80-4cd6-a12a-c22a5ada9600","Type":"ContainerDied","Data":"084074f64499404ac5bc84df3061b67e3901aa6768d0f3f937846799cfcdea67"}
Nov 03 22:00:20.779884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.779714    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"
Nov 03 22:00:20.780434 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.780299    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv" event={"ID":"6175aca6-6a81-4be9-9e6e-9678fe482481","Type":"ContainerDied","Data":"ee785e4d9f41e978bfa3cd88d7f42f825d2db69b147ee6d6694d63456c91d838"}
Nov 03 22:00:20.784960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.784920    5533 scope.go:122] "RemoveContainer" containerID="5f10dc3b30abac8bbf9b0a8f3ebf6cf15feda9b24b370309d33bcbc299aedfb6"
Nov 03 22:00:20.785859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.785745    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv" event={"ID":"6ffe7203-bef8-44df-b103-807099ab28f2","Type":"ContainerDied","Data":"028f45f15039d6cad27bb6db8e615fc0234435bcd9e47459fe15d81c8de0b464"}
Nov 03 22:00:20.785859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.785803    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"
Nov 03 22:00:20.787943 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.787926    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\" (UniqueName: \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\") pod \"6ffe7203-bef8-44df-b103-807099ab28f2\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") "
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.788144    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\") pod \"6ffe7203-bef8-44df-b103-807099ab28f2\" (UID: \"6ffe7203-bef8-44df-b103-807099ab28f2\") "
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.788196    5533 generic.go:356] "Generic (PLEG): container finished" podID="c193516f-b5cc-457c-a205-d9a869bd4241" containerID="bac73204e06bea890be5d01d2c2afd3dc20882787b2ab2cec91c792a726842c4" exitCode=60
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.788273    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerDied","Data":"bac73204e06bea890be5d01d2c2afd3dc20882787b2ab2cec91c792a726842c4"}
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.788606    5533 scope.go:122] "RemoveContainer" containerID="bac73204e06bea890be5d01d2c2afd3dc20882787b2ab2cec91c792a726842c4"
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.788851    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca" pod "6ffe7203-bef8-44df-b103-807099ab28f2" (UID: "6ffe7203-bef8-44df-b103-807099ab28f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.789265 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:20.788944    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg_odh-ai-gateway-infra(c193516f-b5cc-457c-a205-d9a869bd4241)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" podUID="c193516f-b5cc-457c-a205-d9a869bd4241"
Nov 03 22:00:20.791042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.790278    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh" event={"ID":"24c96786-1d79-42f7-b5dd-2a666768d64c","Type":"ContainerDied","Data":"44dbafe245d379edc937de652be4649835b2f0c27492e57ef28f47417a691412"}
Nov 03 22:00:20.791042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.790375    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"
Nov 03 22:00:20.791042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.791009    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd" pod "6ffe7203-bef8-44df-b103-807099ab28f2" (UID: "6ffe7203-bef8-44df-b103-807099ab28f2"). InnerVolumeSpecName "kube-api-access-456rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:20.797822 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.797803    5533 scope.go:122] "RemoveContainer" containerID="e61f99e55eb400b73df9c7cc787b7bd7833c00c2d85ca6ed780d18340c6a9e69"
Nov 03 22:00:20.807385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.807366    5533 scope.go:122] "RemoveContainer" containerID="1843fb17eddacf63444dd05113e46d66e9690922e71237040c0f1cc4dee4d7c5"
Nov 03 22:00:20.816472 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.816449    5533 scope.go:122] "RemoveContainer" containerID="fcb63a3f7c35ffe1224b36aaf9a887b78bb8cccbb6723885428c9a13c3981716"
Nov 03 22:00:20.829862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.829843    5533 scope.go:122] "RemoveContainer" containerID="4712d5f80107db98f415f1e9e6b694f61e7308da3310ee48adff2d276f9e39ac"
Nov 03 22:00:20.892925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.889771    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe7203-bef8-44df-b103-807099ab28f2-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.892925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.889803    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456rd\" (UniqueName: \"kubernetes.io/projected/6ffe7203-bef8-44df-b103-807099ab28f2-kube-api-access-456rd\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.892925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.891535    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:20.900841 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.900803    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-791c32-29895720-wdwdp"]
Nov 03 22:00:20.936884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.936857    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:20.990969 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.990937    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\" (UniqueName: \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\") pod \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") "
Nov 03 22:00:20.991128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.991070    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\") pod \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\" (UID: \"5153f1a0-7a0e-44b4-acc9-578f874f06cd\") "
Nov 03 22:00:20.991398 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.991376    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca" pod "5153f1a0-7a0e-44b4-acc9-578f874f06cd" (UID: "5153f1a0-7a0e-44b4-acc9-578f874f06cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.991743 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.991723    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5153f1a0-7a0e-44b4-acc9-578f874f06cd-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:20.993061 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:20.993042    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx" pod "5153f1a0-7a0e-44b4-acc9-578f874f06cd" (UID: "5153f1a0-7a0e-44b4-acc9-578f874f06cd"). InnerVolumeSpecName "kube-api-access-t48kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:21.092400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.092336    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48kx\" (UniqueName: \"kubernetes.io/projected/5153f1a0-7a0e-44b4-acc9-578f874f06cd-kube-api-access-t48kx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:21.323804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.323767    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:21.328423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.328397    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a41ad0-29895720-7mbwh"]
Nov 03 22:00:21.539660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.539600    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:21.542933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.542903    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0dc66-29895720-vwv4l"]
Nov 03 22:00:21.645888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.645862    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:21.649164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.649138    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a1c634-29895720-hgtkv"]
Nov 03 22:00:21.684169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.684141    5533 scope.go:122] "RemoveContainer" containerID="be30da01cea0b5342d7659885e453d6045940d548f9c6f20010b6b1fc7527b36"
Nov 03 22:00:21.760317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.760274    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:21.766372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.766334    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e84357-29895720-zkngv"]
Nov 03 22:00:21.799916 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.799857    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"
Nov 03 22:00:21.799916 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.799888    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb" event={"ID":"5153f1a0-7a0e-44b4-acc9-578f874f06cd","Type":"ContainerDied","Data":"112048949c32679bfb3f95e75913ea696d4268202c63447617953732ba0a8a7c"}
Nov 03 22:00:21.800094 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.799983    5533 scope.go:122] "RemoveContainer" containerID="36c66582d8e5dfcbae27454a901daeb4082e2488905a24af4086e761f30d93c3"
Nov 03 22:00:21.972053 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.972022    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:21.976044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:21.976015    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-714979-29895720-c7xrb"]
Nov 03 22:00:22.495917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.495884    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" path="/var/lib/kubelet/pods/24c96786-1d79-42f7-b5dd-2a666768d64c/volumes"
Nov 03 22:00:22.496355 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.496339    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" path="/var/lib/kubelet/pods/26ff8a62-56b4-47af-be01-09eb41036b4d/volumes"
Nov 03 22:00:22.496786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.496771    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" path="/var/lib/kubelet/pods/5153f1a0-7a0e-44b4-acc9-578f874f06cd/volumes"
Nov 03 22:00:22.497517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.497500    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" path="/var/lib/kubelet/pods/6175aca6-6a81-4be9-9e6e-9678fe482481/volumes"
Nov 03 22:00:22.497938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.497924    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" path="/var/lib/kubelet/pods/6ffe7203-bef8-44df-b103-807099ab28f2/volumes"
Nov 03 22:00:22.498324 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.498311    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" path="/var/lib/kubelet/pods/cf4003a9-ed80-4cd6-a12a-c22a5ada9600/volumes"
Nov 03 22:00:22.876141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:22.876054    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:23.030918 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.030893    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:23.115265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.115233    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\") pod \"c193516f-b5cc-457c-a205-d9a869bd4241\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") "
Nov 03 22:00:23.115437 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.115345    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\" (UniqueName: \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\") pod \"c193516f-b5cc-457c-a205-d9a869bd4241\" (UID: \"c193516f-b5cc-457c-a205-d9a869bd4241\") "
Nov 03 22:00:23.115589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.115568    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca" pod "c193516f-b5cc-457c-a205-d9a869bd4241" (UID: "c193516f-b5cc-457c-a205-d9a869bd4241"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:23.115713 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.115697    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c193516f-b5cc-457c-a205-d9a869bd4241-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:23.117494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.117469    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg" pod "c193516f-b5cc-457c-a205-d9a869bd4241" (UID: "c193516f-b5cc-457c-a205-d9a869bd4241"). InnerVolumeSpecName "kube-api-access-54hkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:23.217062 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.217034    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54hkg\" (UniqueName: \"kubernetes.io/projected/c193516f-b5cc-457c-a205-d9a869bd4241-kube-api-access-54hkg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:00:23.819384 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.819355    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"
Nov 03 22:00:23.819561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.819392    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg" event={"ID":"c193516f-b5cc-457c-a205-d9a869bd4241","Type":"ContainerDied","Data":"19325db7b3e6ee0d9284a52d981c1f08107c473b1de4b981eb8268a9531da5f4"}
Nov 03 22:00:23.819561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.819453    5533 scope.go:122] "RemoveContainer" containerID="bac73204e06bea890be5d01d2c2afd3dc20882787b2ab2cec91c792a726842c4"
Nov 03 22:00:23.978906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:23.978874    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:24.023469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:24.023440    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9818d8-29895720-gkcvg"]
Nov 03 22:00:24.497025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:24.496993    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" path="/var/lib/kubelet/pods/c193516f-b5cc-457c-a205-d9a869bd4241/volumes"
Nov 03 22:00:26.057344 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:26.057310    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"]
Nov 03 22:00:26.103769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:26.103732    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0944058-s96d2"]
Nov 03 22:00:26.519298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:26.514334    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a029610-2b74-42f2-8076-b6034c24a570" path="/var/lib/kubelet/pods/0a029610-2b74-42f2-8076-b6034c24a570/volumes"
Nov 03 22:00:46.988908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.988867    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:00:46.989778 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989754    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.989778 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989778    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989800    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989808    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989818    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989825    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989836    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989843    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989857    5533 cpu_manager.go:414] "containerMap: removing container" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989864    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989881    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989889    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989900    5533 cpu_manager.go:414] "containerMap: removing container" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989907    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989923    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989930    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989941    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989948    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989959    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989966    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989977    5533 cpu_manager.go:414] "containerMap: removing container" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989983    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.989985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989992    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.989999    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990018    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990024    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990033    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990040    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990056    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990063    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990072    5533 cpu_manager.go:414] "containerMap: removing container" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990078    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990089    5533 cpu_manager.go:414] "containerMap: removing container" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990095    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990106    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990112    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990123    5533 cpu_manager.go:414] "containerMap: removing container" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990130    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990141    5533 cpu_manager.go:414] "containerMap: removing container" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990149    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990161    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990167    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990314    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990330    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990339    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990347    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990356    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990366    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990374    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990384    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990392    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990400    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990408    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990417    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990425    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990435    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990448    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990456    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990465    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990474    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c488d1aa-d8a3-4a7d-b041-1ad5acbd7f31" containerName="cleanup"
Nov 03 22:00:46.990767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.990482    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:00:46.994568 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:46.994551    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.046405 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.046367    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"openshift-service-ca.crt\""
Nov 03 22:00:47.047909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.047877    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:00:47.081224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081196    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.081380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081259    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.081450 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081415    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.081516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081476    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.081576 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081552    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.081622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.081602    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngr75\" (UniqueName: \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.090797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.090772    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"kube-root-ca.crt\""
Nov 03 22:00:47.090899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.090800    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"rate-test-model-714979-kserve-self-signed-certs\""
Nov 03 22:00:47.090964 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.090949    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-714979\"/\"default-dockercfg-7hhh4\""
Nov 03 22:00:47.185257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.185218    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.185584 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.185561    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.186228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.186205    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.186765 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.186741    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.187384 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.185960    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.187384 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.186660    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.190522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.190499    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.190854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.190825    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngr75\" (UniqueName: \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.191294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.191271    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.191593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.191573    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.199650 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.193890    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.251127 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.251056    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngr75\" (UniqueName: \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\") pod \"rate-test-model-714979-kserve-7769cd9d65-lg7hf\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") " pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.311498 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.311469    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:47.691726 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:47.689591    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:00:48.026600 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:48.020067    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" event={"ID":"af25ef86-b58a-4835-99f7-1fb7215c69aa","Type":"ContainerStarted","Data":"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"}
Nov 03 22:00:48.026600 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:48.020168    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" event={"ID":"af25ef86-b58a-4835-99f7-1fb7215c69aa","Type":"ContainerStarted","Data":"18b12b1b499df4c2acc81ad26c21a372dd16c1fcad46a85d4b006508877a312e"}
Nov 03 22:00:48.135332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:48.135276    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" podStartSLOduration=2.135262862 podStartE2EDuration="2.135262862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:48.132078449 +0000 UTC m=+4349.763391926" watchObservedRunningTime="2026-11-03 22:00:48.135262862 +0000 UTC m=+4349.766576339"
Nov 03 22:00:48.312728 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:48.312626    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:48.665855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:48.665773    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:00:49.261738 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:00:49.261695    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:00:51.078535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:51.078492    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"]
Nov 03 22:00:51.104719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:51.104682    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecb4e1a3-xq4bb"]
Nov 03 22:00:52.496039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:52.496005    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3794b91-e9a3-4d98-bf47-a0003d7b0206" path="/var/lib/kubelet/pods/b3794b91-e9a3-4d98-bf47-a0003d7b0206/volumes"
Nov 03 22:00:57.312532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:57.312494    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:57.325571 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:57.325549    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:58.143760 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:58.143729    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:00:59.061894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.060995    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": read tcp 10.133.0.2:42698->10.133.1.52:15021: read: connection reset by peer" start-of-body=
Nov 03 22:00:59.061894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.061086    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": read tcp 10.133.0.2:42698->10.133.1.52:15021: read: connection reset by peer"
Nov 03 22:00:59.141320 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.141294    5533 generic.go:356] "Generic (PLEG): container finished" podID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerID="f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004" exitCode=0
Nov 03 22:00:59.141453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.141366    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerDied","Data":"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"}
Nov 03 22:00:59.141747 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.141732    5533 scope.go:122] "RemoveContainer" containerID="f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"
Nov 03 22:00:59.143208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.143182    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:00:59.143282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.143247    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:00:59.143282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:00:59.143276    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:01:00.150298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:00.150250    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerStarted","Data":"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"}
Nov 03 22:01:00.690960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:00.690928    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:01:00.691168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:00.691040    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body=
Nov 03 22:01:00.691168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:00.691091    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused"
Nov 03 22:01:01.690680 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:01.690647    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body=
Nov 03 22:01:01.691126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:01.690697    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused"
Nov 03 22:01:02.690596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:02.690560    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body=
Nov 03 22:01:02.690882 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:02.690854    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused"
Nov 03 22:01:03.691774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:03.691736    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body=
Nov 03 22:01:03.692274 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:03.691826    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused"
Nov 03 22:01:04.690727 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:04.690689    5533 patch_prober.go:29] interesting pod/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body=
Nov 03 22:01:04.690909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:04.690756    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused"
Nov 03 22:01:05.694430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:05.694399    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:01:05.694937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:05.694913    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:01:05.695723 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:05.695702    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:01:19.263684 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:01:19.263619    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:01:29.042187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:01:29.042152    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:01:49.263973 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:01:49.263938    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:02:04.340594 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.340518    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"]
Nov 03 22:02:04.341118 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341098    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:02:04.341118 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341115    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341129    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341134    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341145    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341150    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341162    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341167    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341179    5533 cpu_manager.go:414] "containerMap: removing container" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341183    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341191    5533 cpu_manager.go:414] "containerMap: removing container" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:02:04.341277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341195    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341292    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ff8a62-56b4-47af-be01-09eb41036b4d" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341303    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c193516f-b5cc-457c-a205-d9a869bd4241" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341317    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6175aca6-6a81-4be9-9e6e-9678fe482481" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341329    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c96786-1d79-42f7-b5dd-2a666768d64c" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341341    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5153f1a0-7a0e-44b4-acc9-578f874f06cd" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341354    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="955621e8-ca4c-476c-af40-b8f3186943b0" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341365    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4003a9-ed80-4cd6-a12a-c22a5ada9600" containerName="cleanup"
Nov 03 22:02:04.341602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.341377    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffe7203-bef8-44df-b103-807099ab28f2" containerName="cleanup"
Nov 03 22:02:04.345385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.345367    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.394054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.394030    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.394234 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.394211    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425vg\" (UniqueName: \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.394335 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.394318    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.400669 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.400630    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"]
Nov 03 22:02:04.443595 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.443574    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:02:04.443746 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.443600    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:04.495442 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.495408    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425vg\" (UniqueName: \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.495608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.495472    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.495608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.495524    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.496227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.496201    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.497818 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.497792    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.554563 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.554537    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425vg\" (UniqueName: \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\") pod \"maas-api-revoke-sub-91741fe2-lrbtv\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:04.660015 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:04.659925    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:05.029931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:05.029892    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"]
Nov 03 22:02:05.034812 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:02:05.034785    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84426868_88e2_449c_8b1b_0a9568d586ef.slice/crio-fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410 WatchSource:0}: Error finding container fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410: Status 404 returned error can't find the container with id fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410
Nov 03 22:02:05.659309 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:05.659231    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv" event={"ID":"84426868-88e2-449c-8b1b-0a9568d586ef","Type":"ContainerStarted","Data":"070909469fce4af1d97cb110b0c7dc30ef774c52c1b38dd6a40ee11180056da3"}
Nov 03 22:02:05.659309 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:05.659289    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv" event={"ID":"84426868-88e2-449c-8b1b-0a9568d586ef","Type":"ContainerStarted","Data":"fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410"}
Nov 03 22:02:05.769754 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:05.769709    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv" podStartSLOduration=1.769696588 podStartE2EDuration="1.769696588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:05.768370911 +0000 UTC m=+4427.399684389" watchObservedRunningTime="2026-11-03 22:02:05.769696588 +0000 UTC m=+4427.401010065"
Nov 03 22:02:06.667422 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:06.667389    5533 generic.go:356] "Generic (PLEG): container finished" podID="84426868-88e2-449c-8b1b-0a9568d586ef" containerID="070909469fce4af1d97cb110b0c7dc30ef774c52c1b38dd6a40ee11180056da3" exitCode=0
Nov 03 22:02:06.667863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:06.667475    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv" event={"ID":"84426868-88e2-449c-8b1b-0a9568d586ef","Type":"ContainerDied","Data":"070909469fce4af1d97cb110b0c7dc30ef774c52c1b38dd6a40ee11180056da3"}
Nov 03 22:02:07.814490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.814464    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:07.936318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.936238    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\" (UniqueName: \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\") pod \"84426868-88e2-449c-8b1b-0a9568d586ef\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") "
Nov 03 22:02:07.936481 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.936435    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\") pod \"84426868-88e2-449c-8b1b-0a9568d586ef\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") "
Nov 03 22:02:07.936558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.936483    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\") pod \"84426868-88e2-449c-8b1b-0a9568d586ef\" (UID: \"84426868-88e2-449c-8b1b-0a9568d586ef\") "
Nov 03 22:02:07.936935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.936898    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca" pod "84426868-88e2-449c-8b1b-0a9568d586ef" (UID: "84426868-88e2-449c-8b1b-0a9568d586ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:07.938292 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.938269    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg" pod "84426868-88e2-449c-8b1b-0a9568d586ef" (UID: "84426868-88e2-449c-8b1b-0a9568d586ef"). InnerVolumeSpecName "kube-api-access-425vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:07.938538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:07.938514    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca" pod "84426868-88e2-449c-8b1b-0a9568d586ef" (UID: "84426868-88e2-449c-8b1b-0a9568d586ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:08.038042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.038009    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:08.038042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.038037    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84426868-88e2-449c-8b1b-0a9568d586ef-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:08.038210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.038075    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425vg\" (UniqueName: \"kubernetes.io/projected/84426868-88e2-449c-8b1b-0a9568d586ef-kube-api-access-425vg\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:08.684658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.684601    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"
Nov 03 22:02:08.684658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.684649    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv" event={"ID":"84426868-88e2-449c-8b1b-0a9568d586ef","Type":"ContainerDied","Data":"fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410"}
Nov 03 22:02:08.684975 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:08.684698    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0b05aac7b054a2635681858a342a4ba57cc3e837bd881450e7d6377a869410"
Nov 03 22:02:15.044018 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.043978    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"]
Nov 03 22:02:15.044596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.044578    5533 cpu_manager.go:414] "containerMap: removing container" podUID="84426868-88e2-449c-8b1b-0a9568d586ef" containerName="revoke-keys"
Nov 03 22:02:15.044670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.044599    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84426868-88e2-449c-8b1b-0a9568d586ef" containerName="revoke-keys"
Nov 03 22:02:15.044816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.044799    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="84426868-88e2-449c-8b1b-0a9568d586ef" containerName="revoke-keys"
Nov 03 22:02:15.047933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.047917    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.098297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.098275    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:15.145593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.145566    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:02:15.149410 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.149385    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"]
Nov 03 22:02:15.215539 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.215514    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.215677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.215660    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbqf\" (UniqueName: \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.215764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.215737    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.316957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.316895    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.317070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.316985    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbqf\" (UniqueName: \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.317070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.317012    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.317672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.317625    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.319128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.319107    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.374131 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.374109    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbqf\" (UniqueName: \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\") pod \"maas-api-revoke-sub-c202087e-7mkj4\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:15.661741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:15.661663    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:16.044994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:16.044963    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"]
Nov 03 22:02:16.749623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:16.749591    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4" event={"ID":"f08c5517-498b-4faa-ba09-dc1ffd2f69a3","Type":"ContainerStarted","Data":"aa2ac85645b4984230e6c9c133e7e70403198f95c7b87ab34e6c603c1f8bc11c"}
Nov 03 22:02:16.749833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:16.749652    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4" event={"ID":"f08c5517-498b-4faa-ba09-dc1ffd2f69a3","Type":"ContainerStarted","Data":"a0a71299add24cc2b3468f2ed540a5f425d600b3e5145e04d2a852af81b90151"}
Nov 03 22:02:16.859981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:16.859929    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4" podStartSLOduration=1.8599162420000002 podStartE2EDuration="1.859916242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:16.857852554 +0000 UTC m=+4438.489166031" watchObservedRunningTime="2026-11-03 22:02:16.859916242 +0000 UTC m=+4438.491229718"
Nov 03 22:02:17.405566 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.405530    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:02:17.409846 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.409827    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.459184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.459153    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c11538-openshift-default-dockercfg-sjr44\""
Nov 03 22:02:17.510129 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.510100    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:02:17.543513 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543481    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543542    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfj7f\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543744 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543692    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543742    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543766    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543853    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.543917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543908    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.544009 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.543994    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.544054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.544043    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645515    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645568    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645598    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfj7f\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645682    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645708    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.645734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645724    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646037 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645808    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646037 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.645878    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646150 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646059    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646155    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646276    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646293    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646410 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646378    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.646667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.646645    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.648286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.648257    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.648385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.648339    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.703495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.703468    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.703724 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.703699    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfj7f\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\") pod \"e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") " pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.723558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.723531    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:17.759806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.759782    5533 generic.go:356] "Generic (PLEG): container finished" podID="f08c5517-498b-4faa-ba09-dc1ffd2f69a3" containerID="aa2ac85645b4984230e6c9c133e7e70403198f95c7b87ab34e6c603c1f8bc11c" exitCode=0
Nov 03 22:02:17.760125 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:17.760099    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4" event={"ID":"f08c5517-498b-4faa-ba09-dc1ffd2f69a3","Type":"ContainerDied","Data":"aa2ac85645b4984230e6c9c133e7e70403198f95c7b87ab34e6c603c1f8bc11c"}
Nov 03 22:02:18.158788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.158754    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:02:18.158788 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:02:18.158782    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922c43e8_fe65_457d_b431_8808db6e293e.slice/crio-632f2f0c3f2084848a6662553ed16c0e4b2f929b81c40068c1be50fc5f143cb7 WatchSource:0}: Error finding container 632f2f0c3f2084848a6662553ed16c0e4b2f929b81c40068c1be50fc5f143cb7: Status 404 returned error can't find the container with id 632f2f0c3f2084848a6662553ed16c0e4b2f929b81c40068c1be50fc5f143cb7
Nov 03 22:02:18.162107 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.162075    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:02:18.162228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.162144    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:02:18.162228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.162175    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:02:18.772492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.772438    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" event={"ID":"922c43e8-fe65-457d-b431-8808db6e293e","Type":"ContainerStarted","Data":"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"}
Nov 03 22:02:18.772993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.772514    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" event={"ID":"922c43e8-fe65-457d-b431-8808db6e293e","Type":"ContainerStarted","Data":"632f2f0c3f2084848a6662553ed16c0e4b2f929b81c40068c1be50fc5f143cb7"}
Nov 03 22:02:18.903695 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.887697    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podStartSLOduration=1.88767609 podStartE2EDuration="1.88767609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:18.885596158 +0000 UTC m=+4440.516909644" watchObservedRunningTime="2026-11-03 22:02:18.88767609 +0000 UTC m=+4440.518989570"
Nov 03 22:02:18.995361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:18.995333    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:19.181658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.180093    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\") pod \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") "
Nov 03 22:02:19.181658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.180296    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\") pod \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") "
Nov 03 22:02:19.181658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.180440    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\" (UniqueName: \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\") pod \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\" (UID: \"f08c5517-498b-4faa-ba09-dc1ffd2f69a3\") "
Nov 03 22:02:19.182982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.182815    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca" pod "f08c5517-498b-4faa-ba09-dc1ffd2f69a3" (UID: "f08c5517-498b-4faa-ba09-dc1ffd2f69a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:19.185494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.185465    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf" pod "f08c5517-498b-4faa-ba09-dc1ffd2f69a3" (UID: "f08c5517-498b-4faa-ba09-dc1ffd2f69a3"). InnerVolumeSpecName "kube-api-access-mlbqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:19.187771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.187736    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca" pod "f08c5517-498b-4faa-ba09-dc1ffd2f69a3" (UID: "f08c5517-498b-4faa-ba09-dc1ffd2f69a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:19.289411 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.289108    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:19.289607 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.289430    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbqf\" (UniqueName: \"kubernetes.io/projected/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-kube-api-access-mlbqf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:19.289607 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.289446    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08c5517-498b-4faa-ba09-dc1ffd2f69a3-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:19.300551 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:02:19.300514    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:02:19.723968 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.723918    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:19.725285 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.725258    5533 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 03 22:02:19.725436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.725312    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 03 22:02:19.792767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.792733    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"
Nov 03 22:02:19.793440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.792730    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4" event={"ID":"f08c5517-498b-4faa-ba09-dc1ffd2f69a3","Type":"ContainerDied","Data":"a0a71299add24cc2b3468f2ed540a5f425d600b3e5145e04d2a852af81b90151"}
Nov 03 22:02:19.793440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:19.792882    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a71299add24cc2b3468f2ed540a5f425d600b3e5145e04d2a852af81b90151"
Nov 03 22:02:20.724587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:20.724549    5533 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 03 22:02:20.724777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:20.724611    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 03 22:02:21.724469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:21.724436    5533 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 03 22:02:21.724952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:21.724496    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 03 22:02:21.775659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:21.774775    5533 scope.go:122] "RemoveContainer" containerID="35ef6d395e426f7aa4ebada67ee96251575e70aa6c2620437a9bbbf86fca9a78"
Nov 03 22:02:21.796460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:21.796436    5533 scope.go:122] "RemoveContainer" containerID="2d9aead31e4e86314ca6a3b02a3341bbadf4debc6485677506430e0c541e192d"
Nov 03 22:02:22.724814 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:22.724775    5533 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 03 22:02:22.725189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:22.724833    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 03 22:02:23.724207 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:23.724174    5533 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Nov 03 22:02:23.724395 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:23.724224    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Nov 03 22:02:24.781606 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:24.781572    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:24.782563 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:24.782532    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:24.783458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:24.783436    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:02:25.468921 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:25.468868    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:02:26.596943 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.596907    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"]
Nov 03 22:02:26.597687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.597668    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f08c5517-498b-4faa-ba09-dc1ffd2f69a3" containerName="revoke-keys"
Nov 03 22:02:26.597687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.597688    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08c5517-498b-4faa-ba09-dc1ffd2f69a3" containerName="revoke-keys"
Nov 03 22:02:26.597816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.597806    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08c5517-498b-4faa-ba09-dc1ffd2f69a3" containerName="revoke-keys"
Nov 03 22:02:26.602616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.602600    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.652378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.652353    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:26.656448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.656421    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"]
Nov 03 22:02:26.697897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.697873    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:02:26.704682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.704659    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.704766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.704699    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wg8\" (UniqueName: \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.704908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.704887    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.805530 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.805503    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.805660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.805550    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.805660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.805586    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wg8\" (UniqueName: \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.806236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.806213    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.808017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.807993    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.864543 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.864488    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wg8\" (UniqueName: \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\") pod \"maas-api-revoke-sub-86089293-r5cfx\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:26.917282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:26.917260    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:27.286192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:27.286159    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"]
Nov 03 22:02:27.287298 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:02:27.287267    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ada136_7ec5_4bdf_b2c2_33686cf87adc.slice/crio-57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264 WatchSource:0}: Error finding container 57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264: Status 404 returned error can't find the container with id 57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264
Nov 03 22:02:27.852690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:27.852576    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx" event={"ID":"89ada136-7ec5-4bdf-b2c2-33686cf87adc","Type":"ContainerStarted","Data":"d45f1aad7520498531ce738a2a7e226d54c3621123ccf67c1a9f5b5246827ba1"}
Nov 03 22:02:27.852690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:27.852646    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx" event={"ID":"89ada136-7ec5-4bdf-b2c2-33686cf87adc","Type":"ContainerStarted","Data":"57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264"}
Nov 03 22:02:27.962936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:27.962888    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx" podStartSLOduration=1.962873381 podStartE2EDuration="1.962873381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:27.961219268 +0000 UTC m=+4449.592532742" watchObservedRunningTime="2026-11-03 22:02:27.962873381 +0000 UTC m=+4449.594186854"
Nov 03 22:02:28.861219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:28.861188    5533 generic.go:356] "Generic (PLEG): container finished" podID="89ada136-7ec5-4bdf-b2c2-33686cf87adc" containerID="d45f1aad7520498531ce738a2a7e226d54c3621123ccf67c1a9f5b5246827ba1" exitCode=0
Nov 03 22:02:28.861614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:28.861224    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx" event={"ID":"89ada136-7ec5-4bdf-b2c2-33686cf87adc","Type":"ContainerDied","Data":"d45f1aad7520498531ce738a2a7e226d54c3621123ccf67c1a9f5b5246827ba1"}
Nov 03 22:02:30.014943 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.014914    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:30.141215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.141127    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\" (UniqueName: \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\") pod \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") "
Nov 03 22:02:30.141215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.141173    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\") pod \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") "
Nov 03 22:02:30.141466 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.141232    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\") pod \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\" (UID: \"89ada136-7ec5-4bdf-b2c2-33686cf87adc\") "
Nov 03 22:02:30.141985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.141960    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca" pod "89ada136-7ec5-4bdf-b2c2-33686cf87adc" (UID: "89ada136-7ec5-4bdf-b2c2-33686cf87adc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:30.143816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.143788    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca" pod "89ada136-7ec5-4bdf-b2c2-33686cf87adc" (UID: "89ada136-7ec5-4bdf-b2c2-33686cf87adc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:30.143916 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.143889    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8" pod "89ada136-7ec5-4bdf-b2c2-33686cf87adc" (UID: "89ada136-7ec5-4bdf-b2c2-33686cf87adc"). InnerVolumeSpecName "kube-api-access-w7wg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:30.243296 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.243269    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:30.243296 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.243295    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wg8\" (UniqueName: \"kubernetes.io/projected/89ada136-7ec5-4bdf-b2c2-33686cf87adc-kube-api-access-w7wg8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:30.243457 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.243305    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89ada136-7ec5-4bdf-b2c2-33686cf87adc-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:30.876524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.876413    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"
Nov 03 22:02:30.876524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.876428    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx" event={"ID":"89ada136-7ec5-4bdf-b2c2-33686cf87adc","Type":"ContainerDied","Data":"57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264"}
Nov 03 22:02:30.876524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:30.876476    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57e0a0984844bd42d3c8dbc8518afcc2c46bc6bd67a48113ec7fe658e819a264"
Nov 03 22:02:32.047025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:32.046990    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"]
Nov 03 22:02:32.093493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:32.093462    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1d62bf-277mz"]
Nov 03 22:02:32.494584 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:32.494546    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cdf96d2-67ff-43d3-9faf-b05bf08a2a57" path="/var/lib/kubelet/pods/4cdf96d2-67ff-43d3-9faf-b05bf08a2a57/volumes"
Nov 03 22:02:37.484561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.484519    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"]
Nov 03 22:02:37.485417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.485392    5533 cpu_manager.go:414] "containerMap: removing container" podUID="89ada136-7ec5-4bdf-b2c2-33686cf87adc" containerName="revoke-keys"
Nov 03 22:02:37.485417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.485416    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ada136-7ec5-4bdf-b2c2-33686cf87adc" containerName="revoke-keys"
Nov 03 22:02:37.485612 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.485574    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ada136-7ec5-4bdf-b2c2-33686cf87adc" containerName="revoke-keys"
Nov 03 22:02:37.489395 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.489372    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.539438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.539411    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:37.586655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.586617    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:02:37.589941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.589917    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"]
Nov 03 22:02:37.623806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.623778    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbc6h\" (UniqueName: \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.623932 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.623915    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.624167 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.624140    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.725395 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.725367    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.725521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.725406    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc6h\" (UniqueName: \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.725521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.725497    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.726180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.726154    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.727780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.727761    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.783499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.783429    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc6h\" (UniqueName: \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\") pod \"maas-api-revoke-sub-0344651c-7hgp8\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:37.803341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:37.803313    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:38.179056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:38.179024    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"]
Nov 03 22:02:38.183670 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:02:38.182150    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc01ffb6_be22_4e59_884d_c013f937b610.slice/crio-6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228 WatchSource:0}: Error finding container 6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228: Status 404 returned error can't find the container with id 6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228
Nov 03 22:02:38.936430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:38.936394    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8" event={"ID":"fc01ffb6-be22-4e59-884d-c013f937b610","Type":"ContainerStarted","Data":"6d168117208441456724510ae51a7e0880d7ec2f2ddf3d522dc4afe8203b1f0a"}
Nov 03 22:02:38.936889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:38.936451    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8" event={"ID":"fc01ffb6-be22-4e59-884d-c013f937b610","Type":"ContainerStarted","Data":"6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228"}
Nov 03 22:02:39.045123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:39.045078    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8" podStartSLOduration=2.045066645 podStartE2EDuration="2.045066645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:39.044799651 +0000 UTC m=+4460.676113128" watchObservedRunningTime="2026-11-03 22:02:39.045066645 +0000 UTC m=+4460.676380122"
Nov 03 22:02:39.945951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:39.945916    5533 generic.go:356] "Generic (PLEG): container finished" podID="fc01ffb6-be22-4e59-884d-c013f937b610" containerID="6d168117208441456724510ae51a7e0880d7ec2f2ddf3d522dc4afe8203b1f0a" exitCode=0
Nov 03 22:02:39.946360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:39.945992    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8" event={"ID":"fc01ffb6-be22-4e59-884d-c013f937b610","Type":"ContainerDied","Data":"6d168117208441456724510ae51a7e0880d7ec2f2ddf3d522dc4afe8203b1f0a"}
Nov 03 22:02:41.109501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.109476    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:41.169510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.169483    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\") pod \"fc01ffb6-be22-4e59-884d-c013f937b610\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") "
Nov 03 22:02:41.169662 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.169535    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\") pod \"fc01ffb6-be22-4e59-884d-c013f937b610\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") "
Nov 03 22:02:41.169758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.169699    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\" (UniqueName: \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\") pod \"fc01ffb6-be22-4e59-884d-c013f937b610\" (UID: \"fc01ffb6-be22-4e59-884d-c013f937b610\") "
Nov 03 22:02:41.169901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.169874    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca" pod "fc01ffb6-be22-4e59-884d-c013f937b610" (UID: "fc01ffb6-be22-4e59-884d-c013f937b610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:41.170240 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.170218    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:41.171888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.171854    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h" pod "fc01ffb6-be22-4e59-884d-c013f937b610" (UID: "fc01ffb6-be22-4e59-884d-c013f937b610"). InnerVolumeSpecName "kube-api-access-rbc6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:41.171888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.171859    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca" pod "fc01ffb6-be22-4e59-884d-c013f937b610" (UID: "fc01ffb6-be22-4e59-884d-c013f937b610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:41.271521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.271431    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbc6h\" (UniqueName: \"kubernetes.io/projected/fc01ffb6-be22-4e59-884d-c013f937b610-kube-api-access-rbc6h\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:41.271521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.271473    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc01ffb6-be22-4e59-884d-c013f937b610-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:02:41.966009 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.965980    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"
Nov 03 22:02:41.966214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.966018    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8" event={"ID":"fc01ffb6-be22-4e59-884d-c013f937b610","Type":"ContainerDied","Data":"6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228"}
Nov 03 22:02:41.966214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:41.966077    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3d01bc523fd24873d9491255de21b4c8efbaa3a3620826f21abe49de759228"
Nov 03 22:02:49.264997 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:02:49.264956    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:02:49.463585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:02:49.463547    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 22:03:04.484021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.483973    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" podUID="9488aa2b-0f03-492a-af09-fc4c8b704b89" containerName="main" containerID="cri-o://f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6" gracePeriod=45
Nov 03 22:03:04.760109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.760073    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 22:03:04.948322 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948283    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\" (UniqueName: \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948354    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948373    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948428    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948485    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\" (UniqueName: \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948563    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\") pod \"9488aa2b-0f03-492a-af09-fc4c8b704b89\" (UID: \"9488aa2b-0f03-492a-af09-fc4c8b704b89\") "
Nov 03 22:03:04.948804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948617    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:04.948804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948628    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:04.948981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.948824    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:04.949068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.949040    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:04.949142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.949104    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:04.949142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.949120    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:04.950679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.950653    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:04.950870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.950847    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:04.951041 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:04.951022    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss" pod "9488aa2b-0f03-492a-af09-fc4c8b704b89" (UID: "9488aa2b-0f03-492a-af09-fc4c8b704b89"). InnerVolumeSpecName "kube-api-access-rnpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:05.050304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.050239    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9488aa2b-0f03-492a-af09-fc4c8b704b89-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:05.050304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.050265    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9488aa2b-0f03-492a-af09-fc4c8b704b89-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:05.050304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.050276    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnpss\" (UniqueName: \"kubernetes.io/projected/9488aa2b-0f03-492a-af09-fc4c8b704b89-kube-api-access-rnpss\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:05.147056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.147027    5533 generic.go:356] "Generic (PLEG): container finished" podID="9488aa2b-0f03-492a-af09-fc4c8b704b89" containerID="f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6" exitCode=0
Nov 03 22:03:05.147228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.147081    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"
Nov 03 22:03:05.147228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.147106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" event={"ID":"9488aa2b-0f03-492a-af09-fc4c8b704b89","Type":"ContainerDied","Data":"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"}
Nov 03 22:03:05.147228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.147152    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4" event={"ID":"9488aa2b-0f03-492a-af09-fc4c8b704b89","Type":"ContainerDied","Data":"1b4ee237a059a3e03b246ac2209a4ad673e0c8c0b15ca69570f13944971b0963"}
Nov 03 22:03:05.147228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.147170    5533 scope.go:122] "RemoveContainer" containerID="f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"
Nov 03 22:03:05.167448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.166041    5533 scope.go:122] "RemoveContainer" containerID="f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"
Nov 03 22:03:05.167687 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:05.167620    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6\": container with ID starting with f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6 not found: ID does not exist" containerID="f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"
Nov 03 22:03:05.167687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.167672    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6"} err="failed to get container status \"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6\": rpc error: code = NotFound desc = could not find container \"f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6\": container with ID starting with f8697a0185f3726876bc907b9d7f3b3c832c3dadb53a4d38676067e88ed3b4f6 not found: ID does not exist"
Nov 03 22:03:05.308207 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.308132    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 22:03:05.356448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:05.356418    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c11538/sub-test-model-c11538-kserve-5f6dd4f9f5-9mnx4"]
Nov 03 22:03:06.495119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:06.495080    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9488aa2b-0f03-492a-af09-fc4c8b704b89" path="/var/lib/kubelet/pods/9488aa2b-0f03-492a-af09-fc4c8b704b89/volumes"
Nov 03 22:03:09.340247 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:09.340213    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:03:13.636985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:13.636949    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 22:03:13.746398 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:13.746367    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 22:03:18.638024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:18.637983    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" podUID="5cfb8287-8b18-458a-bb52-aff38468ea2d" containerName="payload-processing" containerID="cri-o://3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262" gracePeriod=25
Nov 03 22:03:18.747734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:18.747673    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" podUID="97549b96-4caf-47b2-86ca-a471022a8af1" containerName="payload-pre-processing" containerID="cri-o://e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737" gracePeriod=25
Nov 03 22:03:18.908848 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:18.908790    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 22:03:19.011213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.011188    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 22:03:19.026053 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.026028    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\") pod \"5cfb8287-8b18-458a-bb52-aff38468ea2d\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") "
Nov 03 22:03:19.026213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.026194    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\" (UniqueName: \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\") pod \"5cfb8287-8b18-458a-bb52-aff38468ea2d\" (UID: \"5cfb8287-8b18-458a-bb52-aff38468ea2d\") "
Nov 03 22:03:19.026365 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.026344    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume" pod "5cfb8287-8b18-458a-bb52-aff38468ea2d" (UID: "5cfb8287-8b18-458a-bb52-aff38468ea2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:19.026648 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.026621    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfb8287-8b18-458a-bb52-aff38468ea2d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:19.028515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.028485    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr" pod "5cfb8287-8b18-458a-bb52-aff38468ea2d" (UID: "5cfb8287-8b18-458a-bb52-aff38468ea2d"). InnerVolumeSpecName "kube-api-access-2hqzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:19.127095 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.127065    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\" (UniqueName: \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\") pod \"97549b96-4caf-47b2-86ca-a471022a8af1\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") "
Nov 03 22:03:19.127226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.127178    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\") pod \"97549b96-4caf-47b2-86ca-a471022a8af1\" (UID: \"97549b96-4caf-47b2-86ca-a471022a8af1\") "
Nov 03 22:03:19.127659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.127607    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqzr\" (UniqueName: \"kubernetes.io/projected/5cfb8287-8b18-458a-bb52-aff38468ea2d-kube-api-access-2hqzr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:19.127848 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.127818    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume" pod "97549b96-4caf-47b2-86ca-a471022a8af1" (UID: "97549b96-4caf-47b2-86ca-a471022a8af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:19.129312 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.129290    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx" pod "97549b96-4caf-47b2-86ca-a471022a8af1" (UID: "97549b96-4caf-47b2-86ca-a471022a8af1"). InnerVolumeSpecName "kube-api-access-mjqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:19.228966 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.228939    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjqdx\" (UniqueName: \"kubernetes.io/projected/97549b96-4caf-47b2-86ca-a471022a8af1-kube-api-access-mjqdx\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:19.228966 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.228963    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97549b96-4caf-47b2-86ca-a471022a8af1-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:19.262324 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.262291    5533 generic.go:356] "Generic (PLEG): container finished" podID="5cfb8287-8b18-458a-bb52-aff38468ea2d" containerID="3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262" exitCode=0
Nov 03 22:03:19.262496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.262338    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"
Nov 03 22:03:19.262496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.262375    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" event={"ID":"5cfb8287-8b18-458a-bb52-aff38468ea2d","Type":"ContainerDied","Data":"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"}
Nov 03 22:03:19.262496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.262447    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4" event={"ID":"5cfb8287-8b18-458a-bb52-aff38468ea2d","Type":"ContainerDied","Data":"b101f2ade9fd602876f87af8ec3db712a86fa0d27253ab1c19b28fc88f6d417a"}
Nov 03 22:03:19.262496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.262484    5533 scope.go:122] "RemoveContainer" containerID="3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"
Nov 03 22:03:19.262784 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:19.262499    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:03:19.264168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.264140    5533 generic.go:356] "Generic (PLEG): container finished" podID="97549b96-4caf-47b2-86ca-a471022a8af1" containerID="e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737" exitCode=0
Nov 03 22:03:19.264260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.264192    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"
Nov 03 22:03:19.264260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.264195    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" event={"ID":"97549b96-4caf-47b2-86ca-a471022a8af1","Type":"ContainerDied","Data":"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"}
Nov 03 22:03:19.264371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.264307    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd" event={"ID":"97549b96-4caf-47b2-86ca-a471022a8af1","Type":"ContainerDied","Data":"ddf11aba2ac962164898e9575479df094b6296089bc1f527eb978a9fb55e5b55"}
Nov 03 22:03:19.275776 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.275755    5533 scope.go:122] "RemoveContainer" containerID="3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"
Nov 03 22:03:19.276044 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:19.276024    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262\": container with ID starting with 3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262 not found: ID does not exist" containerID="3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"
Nov 03 22:03:19.276104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.276053    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262"} err="failed to get container status \"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262\": rpc error: code = NotFound desc = could not find container \"3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262\": container with ID starting with 3dd7a72dea1fec313a20b6936c357b1c386ec5e5efed43dc3a1bc9c10ba49262 not found: ID does not exist"
Nov 03 22:03:19.276104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.276071    5533 scope.go:122] "RemoveContainer" containerID="e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"
Nov 03 22:03:19.286311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.286295    5533 scope.go:122] "RemoveContainer" containerID="e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"
Nov 03 22:03:19.286584 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:19.286565    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737\": container with ID starting with e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737 not found: ID does not exist" containerID="e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"
Nov 03 22:03:19.286657 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.286591    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737"} err="failed to get container status \"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737\": rpc error: code = NotFound desc = could not find container \"e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737\": container with ID starting with e300b80fa3758b7c26df0caf17bcb736fe6c03d90d62b004b489b81a3ff63737 not found: ID does not exist"
Nov 03 22:03:19.328459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.328435    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:03:19.340189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.340169    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:03:19.383070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.383050    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:03:19.394623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.394604    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:03:19.437049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.437025    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 22:03:19.444282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.444262    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-85db4aad-6c54fcbff5-rltxd"]
Nov 03 22:03:19.546752 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.546685    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 22:03:19.554657 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:19.554623    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85db4aad-5b6695d954-wttx4"]
Nov 03 22:03:20.494242 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:20.494208    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfb8287-8b18-458a-bb52-aff38468ea2d" path="/var/lib/kubelet/pods/5cfb8287-8b18-458a-bb52-aff38468ea2d/volumes"
Nov 03 22:03:20.494677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:20.494624    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97549b96-4caf-47b2-86ca-a471022a8af1" path="/var/lib/kubelet/pods/97549b96-4caf-47b2-86ca-a471022a8af1/volumes"
Nov 03 22:03:22.514682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:22.514630    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 22:03:26.855054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:26.855018    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 22:03:31.856110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:31.856051    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" podUID="2eed1179-c83c-4492-86e5-3388c404067a" containerName="payload-pre-processing" containerID="cri-o://f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97" gracePeriod=25
Nov 03 22:03:32.125181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.125123    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 22:03:32.278507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.278473    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\") pod \"2eed1179-c83c-4492-86e5-3388c404067a\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") "
Nov 03 22:03:32.278692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.278578    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\") pod \"2eed1179-c83c-4492-86e5-3388c404067a\" (UID: \"2eed1179-c83c-4492-86e5-3388c404067a\") "
Nov 03 22:03:32.278957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.278925    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume" pod "2eed1179-c83c-4492-86e5-3388c404067a" (UID: "2eed1179-c83c-4492-86e5-3388c404067a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:32.280790 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.280762    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp" pod "2eed1179-c83c-4492-86e5-3388c404067a" (UID: "2eed1179-c83c-4492-86e5-3388c404067a"). InnerVolumeSpecName "kube-api-access-8htmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:32.360777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.360749    5533 generic.go:356] "Generic (PLEG): container finished" podID="2eed1179-c83c-4492-86e5-3388c404067a" containerID="f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97" exitCode=0
Nov 03 22:03:32.360909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.360804    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"
Nov 03 22:03:32.360909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.360834    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" event={"ID":"2eed1179-c83c-4492-86e5-3388c404067a","Type":"ContainerDied","Data":"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"}
Nov 03 22:03:32.360909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.360886    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl" event={"ID":"2eed1179-c83c-4492-86e5-3388c404067a","Type":"ContainerDied","Data":"166bbd5ad456427afdc76fd75ee8d1c223909e9878951b7008ec11c4e639856a"}
Nov 03 22:03:32.360909 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.360908    5533 scope.go:122] "RemoveContainer" containerID="f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"
Nov 03 22:03:32.370751 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.370731    5533 scope.go:122] "RemoveContainer" containerID="f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"
Nov 03 22:03:32.370995 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:32.370977    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97\": container with ID starting with f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97 not found: ID does not exist" containerID="f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"
Nov 03 22:03:32.371059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.371002    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97"} err="failed to get container status \"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97\": rpc error: code = NotFound desc = could not find container \"f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97\": container with ID starting with f0576083c2022b6d0e2fcf5387de5040827093260cec67e33355f0779f505a97 not found: ID does not exist"
Nov 03 22:03:32.380183 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.380125    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eed1179-c83c-4492-86e5-3388c404067a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:32.380183 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.380154    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htmp\" (UniqueName: \"kubernetes.io/projected/2eed1179-c83c-4492-86e5-3388c404067a-kube-api-access-8htmp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:32.516585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.516561    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 22:03:32.521233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:32.521212    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c11538-5f55f94787-tsgtl"]
Nov 03 22:03:34.493475 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:34.493378    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eed1179-c83c-4492-86e5-3388c404067a" path="/var/lib/kubelet/pods/2eed1179-c83c-4492-86e5-3388c404067a/volumes"
Nov 03 22:03:37.534462 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.534419    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" podUID="6d0819e6-1fac-476c-8c52-07dec75e9b73" containerName="main" containerID="cri-o://d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55" gracePeriod=45
Nov 03 22:03:37.795922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.795868    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 22:03:37.938384 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938357    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938555 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938542    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938567    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938607    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\" (UniqueName: \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938659    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938677    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:37.938967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938704    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\") pod \"6d0819e6-1fac-476c-8c52-07dec75e9b73\" (UID: \"6d0819e6-1fac-476c-8c52-07dec75e9b73\") "
Nov 03 22:03:37.938967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.938820    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:37.939126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.939057    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:37.939126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.939082    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:37.939301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.939278    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:37.941259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.941233    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:37.941383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.941281    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:37.941699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:37.941680    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c" pod "6d0819e6-1fac-476c-8c52-07dec75e9b73" (UID: "6d0819e6-1fac-476c-8c52-07dec75e9b73"). InnerVolumeSpecName "kube-api-access-lgb8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:38.039826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.039801    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:38.039826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.039825    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgb8c\" (UniqueName: \"kubernetes.io/projected/6d0819e6-1fac-476c-8c52-07dec75e9b73-kube-api-access-lgb8c\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:38.039987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.039836    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d0819e6-1fac-476c-8c52-07dec75e9b73-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:38.039987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.039844    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d0819e6-1fac-476c-8c52-07dec75e9b73-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:38.404172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.404143    5533 generic.go:356] "Generic (PLEG): container finished" podID="6d0819e6-1fac-476c-8c52-07dec75e9b73" containerID="d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55" exitCode=0
Nov 03 22:03:38.404338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.404187    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"
Nov 03 22:03:38.404338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.404227    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" event={"ID":"6d0819e6-1fac-476c-8c52-07dec75e9b73","Type":"ContainerDied","Data":"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"}
Nov 03 22:03:38.404338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.404270    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn" event={"ID":"6d0819e6-1fac-476c-8c52-07dec75e9b73","Type":"ContainerDied","Data":"202634cdccc529287e7ed9b5c7674e817ae3de78d965d59a7b290052cf263973"}
Nov 03 22:03:38.404338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.404291    5533 scope.go:122] "RemoveContainer" containerID="d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"
Nov 03 22:03:38.414963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.414948    5533 scope.go:122] "RemoveContainer" containerID="d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"
Nov 03 22:03:38.415214 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:38.415198    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55\": container with ID starting with d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55 not found: ID does not exist" containerID="d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"
Nov 03 22:03:38.415263 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.415219    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55"} err="failed to get container status \"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55\": rpc error: code = NotFound desc = could not find container \"d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55\": container with ID starting with d75286d231107481d5f05a6a79aa20f092cff241ebc70c811e4a072a3bbeed55 not found: ID does not exist"
Nov 03 22:03:38.559480 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.559453    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 22:03:38.606280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:38.606250    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0dc66/sub-test-model-f0dc66-kserve-8fd645555-rgzxn"]
Nov 03 22:03:39.241052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.241018    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 22:03:39.241277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.241240    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" containerID="cri-o://bb80602f0234f9e7a852056e160b600c923e0a4cc04cbc9520e9114a8ebf81c1" gracePeriod=30
Nov 03 22:03:39.288081 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.288052    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 22:03:39.288280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.288257    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" containerID="cri-o://8efc8eaef5a28dfdb6634b63ef36796df870b859f363d1f6a74c09db5321217d" gracePeriod=30
Nov 03 22:03:39.291001 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.290971    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 22:03:39.291172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.291148    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy" containerID="cri-o://c354991d52b2fddd747aa54a8daee909d8cff1cb11fc157cd5ee74574593260d" gracePeriod=30
Nov 03 22:03:39.334735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.334702    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 22:03:39.334911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.334889    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy" containerID="cri-o://38daf7b7e35dd219872aaf4fd10265f82b5a1c873b29579b7aee6369744395db" gracePeriod=30
Nov 03 22:03:39.607133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.607046    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-99g77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:03:39.607133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:39.607113    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:03:40.494823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:40.494791    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0819e6-1fac-476c-8c52-07dec75e9b73" path="/var/lib/kubelet/pods/6d0819e6-1fac-476c-8c52-07dec75e9b73/volumes"
Nov 03 22:03:44.183946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.183909    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:03:44.184347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.183974    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:03:44.463554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.463036    5533 generic.go:356] "Generic (PLEG): container finished" podID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerID="8efc8eaef5a28dfdb6634b63ef36796df870b859f363d1f6a74c09db5321217d" exitCode=0
Nov 03 22:03:44.463554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.463154    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" event={"ID":"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc","Type":"ContainerDied","Data":"8efc8eaef5a28dfdb6634b63ef36796df870b859f363d1f6a74c09db5321217d"}
Nov 03 22:03:44.465687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.465521    5533 generic.go:356] "Generic (PLEG): container finished" podID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerID="38daf7b7e35dd219872aaf4fd10265f82b5a1c873b29579b7aee6369744395db" exitCode=0
Nov 03 22:03:44.465687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.465600    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" event={"ID":"908a4919-eb96-41ec-8ae7-9c7ea103e288","Type":"ContainerDied","Data":"38daf7b7e35dd219872aaf4fd10265f82b5a1c873b29579b7aee6369744395db"}
Nov 03 22:03:44.468257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.468009    5533 generic.go:356] "Generic (PLEG): container finished" podID="6577519e-ac8f-42bc-81b0-04852356456a" containerID="bb80602f0234f9e7a852056e160b600c923e0a4cc04cbc9520e9114a8ebf81c1" exitCode=0
Nov 03 22:03:44.468257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.468176    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" event={"ID":"6577519e-ac8f-42bc-81b0-04852356456a","Type":"ContainerDied","Data":"bb80602f0234f9e7a852056e160b600c923e0a4cc04cbc9520e9114a8ebf81c1"}
Nov 03 22:03:44.470581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.470473    5533 generic.go:356] "Generic (PLEG): container finished" podID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerID="c354991d52b2fddd747aa54a8daee909d8cff1cb11fc157cd5ee74574593260d" exitCode=0
Nov 03 22:03:44.470581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.470522    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" event={"ID":"2dc6d9ee-99df-4755-95d7-6e5196567e04","Type":"ContainerDied","Data":"c354991d52b2fddd747aa54a8daee909d8cff1cb11fc157cd5ee74574593260d"}
Nov 03 22:03:44.593900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.593873    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 22:03:44.611897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.611870    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.612124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612099    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612147    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612200    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612242    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612272    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612322    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612355    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.612447    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\") pod \"6577519e-ac8f-42bc-81b0-04852356456a\" (UID: \"6577519e-ac8f-42bc-81b0-04852356456a\") "
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.613327    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.613533    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.613533    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.613601    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.613768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.613682    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.615109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.615070    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:44.615801 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.615759    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.615952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.615919    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.616958 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.616936    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk" pod "6577519e-ac8f-42bc-81b0-04852356456a" (UID: "6577519e-ac8f-42bc-81b0-04852356456a"). InnerVolumeSpecName "kube-api-access-rf2sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.644727 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.644699    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 22:03:44.685830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.685255    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 22:03:44.714400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714315    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714360    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714403    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714464    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714511    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714537    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714575    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714604    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.714771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714673    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.715252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714786    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.715252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714813    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.715252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714842    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.715252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714873    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.715252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.714911    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.716326 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.715877    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.716326 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.716127    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.716326 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.716251    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.716587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.716383    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:44.717421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.716925    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:44.717421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.717156    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.717421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.717241    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.717770 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.717732    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718314    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\") pod \"2dc6d9ee-99df-4755-95d7-6e5196567e04\" (UID: \"2dc6d9ee-99df-4755-95d7-6e5196567e04\") "
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718377    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718422    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718457    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\") pod \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\" (UID: \"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc\") "
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718943    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718965    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718979    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc6d9ee-99df-4755-95d7-6e5196567e04-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.718992    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6577519e-ac8f-42bc-81b0-04852356456a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719006    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719019    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719032    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719044    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719056    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2sk\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-kube-api-access-rf2sk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719070    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719090    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719101    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6577519e-ac8f-42bc-81b0-04852356456a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719114    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6577519e-ac8f-42bc-81b0-04852356456a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719125    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6577519e-ac8f-42bc-81b0-04852356456a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719137    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719149    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.719354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.719161    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.723025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.722994    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.723595 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.723552    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.727223 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.727186    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "kube-api-access-q58m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.727908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.727882    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:44.728062 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.728040    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.728182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.728159    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.728304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.728248    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.728374 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.728314    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo" pod "2dc6d9ee-99df-4755-95d7-6e5196567e04" (UID: "2dc6d9ee-99df-4755-95d7-6e5196567e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:44.729333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.729301    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.732026 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.731976    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49" pod "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" (UID: "f71ba5c1-7b5c-419e-88fa-65abf3e22cfc"). InnerVolumeSpecName "kube-api-access-r6x49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.741089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.741061    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 22:03:44.820168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820134    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820226    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820282    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820321    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820385    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820448    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820520    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820765 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820585    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.820765 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.820615    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\") pod \"908a4919-eb96-41ec-8ae7-9c7ea103e288\" (UID: \"908a4919-eb96-41ec-8ae7-9c7ea103e288\") "
Nov 03 22:03:44.821226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821203    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821233    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821249    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821265    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x49\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-kube-api-access-r6x49\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821280    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58m6\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-kube-api-access-q58m6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821295    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821310    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821324    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821339    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc6d9ee-99df-4755-95d7-6e5196567e04-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821354    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.821930 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.821903    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.822125 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.822101    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.822422 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.822400    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:44.822940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.822913    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "kube-api-access-k442j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.823196 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.823167    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.823455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.823434    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.824875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.824853    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:44.825429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.825400    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:44.826659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.826620    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token" pod "908a4919-eb96-41ec-8ae7-9c7ea103e288" (UID: "908a4919-eb96-41ec-8ae7-9c7ea103e288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:44.922460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922423    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908a4919-eb96-41ec-8ae7-9c7ea103e288-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922457    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922471    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922483    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922497    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922509    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922520    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922533    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k442j\" (UniqueName: \"kubernetes.io/projected/908a4919-eb96-41ec-8ae7-9c7ea103e288-kube-api-access-k442j\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:44.922697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:44.922547    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908a4919-eb96-41ec-8ae7-9c7ea103e288-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:45.479994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.479964    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"
Nov 03 22:03:45.480403 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.479973    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z" event={"ID":"908a4919-eb96-41ec-8ae7-9c7ea103e288","Type":"ContainerDied","Data":"70a8dcecc7478e1a5dbc971aafa28a170881a031469446f5bae00b37d844a8b9"}
Nov 03 22:03:45.480403 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.480115    5533 scope.go:122] "RemoveContainer" containerID="38daf7b7e35dd219872aaf4fd10265f82b5a1c873b29579b7aee6369744395db"
Nov 03 22:03:45.481758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.481736    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"
Nov 03 22:03:45.481888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.481775    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw" event={"ID":"6577519e-ac8f-42bc-81b0-04852356456a","Type":"ContainerDied","Data":"c58c431d4b435fb14b6437fbfcf5b24107a9c4c3fdfa0e2274aa5f7a21dc387d"}
Nov 03 22:03:45.487503 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.487464    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5" event={"ID":"2dc6d9ee-99df-4755-95d7-6e5196567e04","Type":"ContainerDied","Data":"f4e6451b0b8f06e19cb33b2d3cc5e160eccc84ff516347d6baab4235837b4413"}
Nov 03 22:03:45.487653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.487559    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"
Nov 03 22:03:45.491285 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.491266    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77" event={"ID":"f71ba5c1-7b5c-419e-88fa-65abf3e22cfc","Type":"ContainerDied","Data":"67b001156105412eca7c88c80b8ba5950e02793f48ae62c436437ae044a7d8b7"}
Nov 03 22:03:45.491285 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.491279    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"
Nov 03 22:03:45.507774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.507751    5533 scope.go:122] "RemoveContainer" containerID="bb80602f0234f9e7a852056e160b600c923e0a4cc04cbc9520e9114a8ebf81c1"
Nov 03 22:03:45.517970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.517928    5533 scope.go:122] "RemoveContainer" containerID="c354991d52b2fddd747aa54a8daee909d8cff1cb11fc157cd5ee74574593260d"
Nov 03 22:03:45.533433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.533415    5533 scope.go:122] "RemoveContainer" containerID="8efc8eaef5a28dfdb6634b63ef36796df870b859f363d1f6a74c09db5321217d"
Nov 03 22:03:45.641852 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.641824    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 22:03:45.687293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.687251    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-tp7d5"]
Nov 03 22:03:45.749626 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.749597    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 22:03:45.752900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.752876    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-4hs5z"]
Nov 03 22:03:45.854425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.854332    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 22:03:45.857987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.857962    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-99g77"]
Nov 03 22:03:45.961598 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.961569    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 22:03:45.965564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:45.965541    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nxpqw"]
Nov 03 22:03:46.505531 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:46.505492    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" path="/var/lib/kubelet/pods/2dc6d9ee-99df-4755-95d7-6e5196567e04/volumes"
Nov 03 22:03:46.506206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:46.506181    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6577519e-ac8f-42bc-81b0-04852356456a" path="/var/lib/kubelet/pods/6577519e-ac8f-42bc-81b0-04852356456a/volumes"
Nov 03 22:03:46.506805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:46.506787    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" path="/var/lib/kubelet/pods/908a4919-eb96-41ec-8ae7-9c7ea103e288/volumes"
Nov 03 22:03:46.507848 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:46.507821    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" path="/var/lib/kubelet/pods/f71ba5c1-7b5c-419e-88fa-65abf3e22cfc/volumes"
Nov 03 22:03:49.262208 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:03:49.262164    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:03:54.699474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:54.699431    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 22:03:54.700014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:54.699728    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy" containerID="cri-o://db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea" gracePeriod=30
Nov 03 22:03:54.990354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:54.990272    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:03:56.383294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383258    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"]
Nov 03 22:03:56.383800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383785    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2eed1179-c83c-4492-86e5-3388c404067a" containerName="payload-pre-processing"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383801    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eed1179-c83c-4492-86e5-3388c404067a" containerName="payload-pre-processing"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383816    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fc01ffb6-be22-4e59-884d-c013f937b610" containerName="revoke-keys"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383822    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc01ffb6-be22-4e59-884d-c013f937b610" containerName="revoke-keys"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383829    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6d0819e6-1fac-476c-8c52-07dec75e9b73" containerName="main"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383834    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0819e6-1fac-476c-8c52-07dec75e9b73" containerName="main"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383842    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383847    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383858    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy"
Nov 03 22:03:56.383863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383863    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383884    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9488aa2b-0f03-492a-af09-fc4c8b704b89" containerName="main"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383891    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9488aa2b-0f03-492a-af09-fc4c8b704b89" containerName="main"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383916    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383921    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383930    5533 cpu_manager.go:414] "containerMap: removing container" podUID="97549b96-4caf-47b2-86ca-a471022a8af1" containerName="payload-pre-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383935    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97549b96-4caf-47b2-86ca-a471022a8af1" containerName="payload-pre-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383942    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5cfb8287-8b18-458a-bb52-aff38468ea2d" containerName="payload-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383946    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfb8287-8b18-458a-bb52-aff38468ea2d" containerName="payload-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383954    5533 cpu_manager.go:414] "containerMap: removing container" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.383972    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384084    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eed1179-c83c-4492-86e5-3388c404067a" containerName="payload-pre-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384097    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc01ffb6-be22-4e59-884d-c013f937b610" containerName="revoke-keys"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384106    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0819e6-1fac-476c-8c52-07dec75e9b73" containerName="main"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384117    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6577519e-ac8f-42bc-81b0-04852356456a" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384123    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71ba5c1-7b5c-419e-88fa-65abf3e22cfc" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384131    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9488aa2b-0f03-492a-af09-fc4c8b704b89" containerName="main"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384138    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc6d9ee-99df-4755-95d7-6e5196567e04" containerName="istio-proxy"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384146    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="97549b96-4caf-47b2-86ca-a471022a8af1" containerName="payload-pre-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384155    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfb8287-8b18-458a-bb52-aff38468ea2d" containerName="payload-processing"
Nov 03 22:03:56.384303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.384161    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="908a4919-eb96-41ec-8ae7-9c7ea103e288" containerName="istio-proxy"
Nov 03 22:03:56.389237 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.389219    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.442244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.442217    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"]
Nov 03 22:03:56.444016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.443980    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.444147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.444031    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.444147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.444115    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99sj\" (UniqueName: \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.484912 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.484885    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:03:56.485215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.485188    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:03:56.545044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.545018    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.545147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.545058    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.545147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.545096    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99sj\" (UniqueName: \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.545692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.545671    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.547445 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.547427    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.602340 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.602315    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99sj\" (UniqueName: \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\") pod \"maas-api-revoke-sub-ca14dd43-ssk7w\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:56.714686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:56.702247    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:57.106682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:57.106631    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"]
Nov 03 22:03:57.108526 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:03:57.108483    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd809bef_5551_416e_a714_72347a4ad232.slice/crio-581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7 WatchSource:0}: Error finding container 581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7: Status 404 returned error can't find the container with id 581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7
Nov 03 22:03:57.585457 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:57.585417    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w" event={"ID":"fd809bef-5551-416e-a714-72347a4ad232","Type":"ContainerStarted","Data":"9171985ac92ad9556e907eb781ee3dc683e37c9a494a383224c91d2fa5c0d972"}
Nov 03 22:03:57.585891 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:57.585479    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w" event={"ID":"fd809bef-5551-416e-a714-72347a4ad232","Type":"ContainerStarted","Data":"581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7"}
Nov 03 22:03:57.695528 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:57.695469    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w" podStartSLOduration=1.695451015 podStartE2EDuration="1.695451015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:57.694170356 +0000 UTC m=+4539.325483833" watchObservedRunningTime="2026-11-03 22:03:57.695451015 +0000 UTC m=+4539.326764493"
Nov 03 22:03:58.593600 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:58.593572    5533 generic.go:356] "Generic (PLEG): container finished" podID="fd809bef-5551-416e-a714-72347a4ad232" containerID="9171985ac92ad9556e907eb781ee3dc683e37c9a494a383224c91d2fa5c0d972" exitCode=0
Nov 03 22:03:58.593994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:58.593674    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w" event={"ID":"fd809bef-5551-416e-a714-72347a4ad232","Type":"ContainerDied","Data":"9171985ac92ad9556e907eb781ee3dc683e37c9a494a383224c91d2fa5c0d972"}
Nov 03 22:03:59.800762 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.800735    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:03:59.891564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.891535    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\") pod \"fd809bef-5551-416e-a714-72347a4ad232\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") "
Nov 03 22:03:59.891758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.891610    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\") pod \"fd809bef-5551-416e-a714-72347a4ad232\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") "
Nov 03 22:03:59.891758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.891665    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\" (UniqueName: \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\") pod \"fd809bef-5551-416e-a714-72347a4ad232\" (UID: \"fd809bef-5551-416e-a714-72347a4ad232\") "
Nov 03 22:03:59.892491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.892465    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca" pod "fd809bef-5551-416e-a714-72347a4ad232" (UID: "fd809bef-5551-416e-a714-72347a4ad232"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:59.898119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.898077    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca" pod "fd809bef-5551-416e-a714-72347a4ad232" (UID: "fd809bef-5551-416e-a714-72347a4ad232"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:03:59.898119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.898115    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj" pod "fd809bef-5551-416e-a714-72347a4ad232" (UID: "fd809bef-5551-416e-a714-72347a4ad232"). InnerVolumeSpecName "kube-api-access-f99sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:59.993170 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.993132    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd809bef-5551-416e-a714-72347a4ad232-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:59.993170 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.993170    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99sj\" (UniqueName: \"kubernetes.io/projected/fd809bef-5551-416e-a714-72347a4ad232-kube-api-access-f99sj\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:59.993361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.993186    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd809bef-5551-416e-a714-72347a4ad232-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:03:59.995621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:03:59.995598    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:04:00.094362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094331    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094377    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094443    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094501    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094581    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094597    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:00.094735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094612    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094674    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094703    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.094981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.094766    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\") pod \"d2343faa-fc11-46e1-8251-9339ae36b00a\" (UID: \"d2343faa-fc11-46e1-8251-9339ae36b00a\") "
Nov 03 22:04:00.095058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095031    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:00.095159 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095137    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:00.095229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095189    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:00.095229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095195    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.095229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095214    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2343faa-fc11-46e1-8251-9339ae36b00a-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.095394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.095303    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:00.096747 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.096717    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "kube-api-access-nzc84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:00.096863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.096776    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:04:00.097104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.097086    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:00.097235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.097215    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy" pod "d2343faa-fc11-46e1-8251-9339ae36b00a" (UID: "d2343faa-fc11-46e1-8251-9339ae36b00a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:00.196760 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196731    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.196760 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196762    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.197032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196777    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.197032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196792    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.197032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196807    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzc84\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-kube-api-access-nzc84\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.197032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196821    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.197032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.196836    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2343faa-fc11-46e1-8251-9339ae36b00a-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:00.280517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.280482    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 22:04:00.289607 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.289575    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 22:04:00.302592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.302566    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:04:00.302934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.302905    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy" containerID="cri-o://bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431" gracePeriod=30
Nov 03 22:04:00.305188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.305163    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 22:04:00.608914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.608885    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"
Nov 03 22:04:00.609080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.608911    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w" event={"ID":"fd809bef-5551-416e-a714-72347a4ad232","Type":"ContainerDied","Data":"581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7"}
Nov 03 22:04:00.609080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.608985    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581d86633501acbc5c1969abf76aa05ecc8648ff2cef3a60fd1c8b4029c073d7"
Nov 03 22:04:00.610690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.610663    5533 generic.go:356] "Generic (PLEG): container finished" podID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerID="db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea" exitCode=0
Nov 03 22:04:00.610843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.610695    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerDied","Data":"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"}
Nov 03 22:04:00.610843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.610736    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"
Nov 03 22:04:00.610843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.610759    5533 scope.go:122] "RemoveContainer" containerID="db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"
Nov 03 22:04:00.610843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.610747    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps" event={"ID":"d2343faa-fc11-46e1-8251-9339ae36b00a","Type":"ContainerDied","Data":"6808e14126b5d4bb97cc62d5de3ac86b125dbb788b89c22d085bff6933daf789"}
Nov 03 22:04:00.621353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.621331    5533 scope.go:122] "RemoveContainer" containerID="f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"
Nov 03 22:04:00.630851 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.630834    5533 scope.go:122] "RemoveContainer" containerID="db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"
Nov 03 22:04:00.631114 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:00.631094    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea\": container with ID starting with db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea not found: ID does not exist" containerID="db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"
Nov 03 22:04:00.631177 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.631119    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea"} err="failed to get container status \"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea\": rpc error: code = NotFound desc = could not find container \"db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea\": container with ID starting with db9686185eb1dd743659ec5a533f841a71172fa54b5b316a50424c1e8c5dc2ea not found: ID does not exist"
Nov 03 22:04:00.631177 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.631135    5533 scope.go:122] "RemoveContainer" containerID="f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"
Nov 03 22:04:00.631368 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:00.631353    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004\": container with ID starting with f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004 not found: ID does not exist" containerID="f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"
Nov 03 22:04:00.631411 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.631372    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004"} err="failed to get container status \"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004\": rpc error: code = NotFound desc = could not find container \"f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004\": container with ID starting with f00f75aaf81af928f0c9bd911e91ea6fca41392863f711ef5e2398938636c004 not found: ID does not exist"
Nov 03 22:04:00.894259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.894168    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 22:04:00.898494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:00.898467    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85db4aad-openshift-default-7d6ccdfc65-zp9ps"]
Nov 03 22:04:02.494742 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:02.494704    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" path="/var/lib/kubelet/pods/d2343faa-fc11-46e1-8251-9339ae36b00a/volumes"
Nov 03 22:04:05.565183 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.565159    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:04:05.647033 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.646955    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647033 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.646995    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647072    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647110    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647256    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:05.647362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647272    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647359    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647379    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647397    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:05.647523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647412    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647443    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\") pod \"922c43e8-fe65-457d-b431-8808db6e293e\" (UID: \"922c43e8-fe65-457d-b431-8808db6e293e\") "
Nov 03 22:04:05.647523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647447    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:05.647806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647654    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:05.647939 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.647918    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:05.649259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.648184    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.649259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.648207    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c43e8-fe65-457d-b431-8808db6e293e-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.649259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.648223    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.649259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.648235    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.649259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.648248    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.651545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.650435    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:05.651545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.650860    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:04:05.651545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.651215    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:05.652092 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.652069    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f" pod "922c43e8-fe65-457d-b431-8808db6e293e" (UID: "922c43e8-fe65-457d-b431-8808db6e293e"). InnerVolumeSpecName "kube-api-access-tfj7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:05.653656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.653312    5533 generic.go:356] "Generic (PLEG): container finished" podID="922c43e8-fe65-457d-b431-8808db6e293e" containerID="bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431" exitCode=0
Nov 03 22:04:05.653656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.653358    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" event={"ID":"922c43e8-fe65-457d-b431-8808db6e293e","Type":"ContainerDied","Data":"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"}
Nov 03 22:04:05.653656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.653367    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"
Nov 03 22:04:05.653656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.653422    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs" event={"ID":"922c43e8-fe65-457d-b431-8808db6e293e","Type":"ContainerDied","Data":"632f2f0c3f2084848a6662553ed16c0e4b2f929b81c40068c1be50fc5f143cb7"}
Nov 03 22:04:05.653656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.653452    5533 scope.go:122] "RemoveContainer" containerID="bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"
Nov 03 22:04:05.683805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.683768    5533 scope.go:122] "RemoveContainer" containerID="bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"
Nov 03 22:04:05.684066 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:05.684046    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431\": container with ID starting with bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431 not found: ID does not exist" containerID="bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"
Nov 03 22:04:05.684134 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.684078    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431"} err="failed to get container status \"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431\": rpc error: code = NotFound desc = could not find container \"bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431\": container with ID starting with bc719d84e2cb9f95bf409edf29ea21b63ee8e069a6e7f5dcba68293d9772e431 not found: ID does not exist"
Nov 03 22:04:05.749133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.749110    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.749231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.749136    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfj7f\" (UniqueName: \"kubernetes.io/projected/922c43e8-fe65-457d-b431-8808db6e293e-kube-api-access-tfj7f\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.749231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.749147    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c43e8-fe65-457d-b431-8808db6e293e-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.749231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.749156    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c43e8-fe65-457d-b431-8808db6e293e-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:05.834069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.834041    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:04:05.838162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:05.838135    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c11538-openshift-default-84b56cd9c7-5lphs"]
Nov 03 22:04:06.494291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:06.494253    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922c43e8-fe65-457d-b431-8808db6e293e" path="/var/lib/kubelet/pods/922c43e8-fe65-457d-b431-8808db6e293e/volumes"
Nov 03 22:04:14.623106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623065    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"]
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623570    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623582    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623595    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623600    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623611    5533 cpu_manager.go:414] "containerMap: removing container" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623617    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623666    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fd809bef-5551-416e-a714-72347a4ad232" containerName="revoke-keys"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623671    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd809bef-5551-416e-a714-72347a4ad232" containerName="revoke-keys"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623748    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="922c43e8-fe65-457d-b431-8808db6e293e" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623770    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd809bef-5551-416e-a714-72347a4ad232" containerName="revoke-keys"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623782    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.623800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.623792    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2343faa-fc11-46e1-8251-9339ae36b00a" containerName="istio-proxy"
Nov 03 22:04:14.628623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.628602    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.678269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.678249    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:04:14.724895 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.724873    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:04:14.730657 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.728315    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"]
Nov 03 22:04:14.737257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.737228    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.737408 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.737383    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.737460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.737433    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.837995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.837971    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.838107 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.838037    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.838107 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.838072    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.838742 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.838718    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.840498 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.840477    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.895455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.895402    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\") pod \"maas-api-revoke-sub-79603805-tv4lk\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:14.943646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:14.943618    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:15.306685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.306479    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main" containerID="cri-o://f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787" gracePeriod=45
Nov 03 22:04:15.311007 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.310976    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"]
Nov 03 22:04:15.312460 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:04:15.312161    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3783ee3b_5f53_4705_b690_20030433d97f.slice/crio-304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932 WatchSource:0}: Error finding container 304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932: Status 404 returned error can't find the container with id 304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932
Nov 03 22:04:15.312460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.312331    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main" containerID="cri-o://08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f" gracePeriod=45
Nov 03 22:04:15.324523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.324435    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main" containerID="cri-o://174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911" gracePeriod=45
Nov 03 22:04:15.597397 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.597369    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 22:04:15.624056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.624027    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 22:04:15.628369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.628344    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 22:04:15.645937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.645911    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\" (UniqueName: \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.646046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646026    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\" (UniqueName: \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646074    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.646136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646107    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.646203 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646138    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646203 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646192    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646325 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646269    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.646325 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646296    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.646453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646338    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.646453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646378    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.646453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646434    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.646664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646484    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.646664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646519    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\" (UniqueName: \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646551    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.646664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646595    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646657    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\") pod \"651af5cf-fbb4-4ade-b468-d1de7b667163\" (UID: \"651af5cf-fbb4-4ade-b468-d1de7b667163\") "
Nov 03 22:04:15.646961 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.646690    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\" (UniqueName: \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\") pod \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\" (UID: \"7df02540-271e-4f6e-bc1d-1778fc2f5f0e\") "
Nov 03 22:04:15.647313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.647288    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.647448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.647432    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.647961 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.647923    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648077 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648944 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648368    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648944 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648397    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648944 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648753    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648944 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648895    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.648944 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.648896    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "kube-api-access-tc2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:15.650307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.650209    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:15.650307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.650227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.650307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.650274    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:15.650694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.650663    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9" pod "7df02540-271e-4f6e-bc1d-1778fc2f5f0e" (UID: "7df02540-271e-4f6e-bc1d-1778fc2f5f0e"). InnerVolumeSpecName "kube-api-access-9w7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:15.650965 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.650939    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:15.651140 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.651112    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.651245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.651227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.651735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.651716    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r" pod "651af5cf-fbb4-4ade-b468-d1de7b667163" (UID: "651af5cf-fbb4-4ade-b468-d1de7b667163"). InnerVolumeSpecName "kube-api-access-74x9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:15.731532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.731491    5533 generic.go:356] "Generic (PLEG): container finished" podID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerID="f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787" exitCode=0
Nov 03 22:04:15.731740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.731686    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"
Nov 03 22:04:15.732226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.732192    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" event={"ID":"7df02540-271e-4f6e-bc1d-1778fc2f5f0e","Type":"ContainerDied","Data":"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"}
Nov 03 22:04:15.732341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.732245    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm" event={"ID":"7df02540-271e-4f6e-bc1d-1778fc2f5f0e","Type":"ContainerDied","Data":"caeb82419bbacace3f8df2dc3bda793ec70c1bde3a66c1a594b5023b5b505818"}
Nov 03 22:04:15.732341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.732269    5533 scope.go:122] "RemoveContainer" containerID="f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"
Nov 03 22:04:15.734440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.734411    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk" event={"ID":"3783ee3b-5f53-4705-b690-20030433d97f","Type":"ContainerStarted","Data":"304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932"}
Nov 03 22:04:15.744871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.742096    5533 generic.go:356] "Generic (PLEG): container finished" podID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerID="08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f" exitCode=0
Nov 03 22:04:15.744871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.742204    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" event={"ID":"651af5cf-fbb4-4ade-b468-d1de7b667163","Type":"ContainerDied","Data":"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"}
Nov 03 22:04:15.744871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.742229    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" event={"ID":"651af5cf-fbb4-4ade-b468-d1de7b667163","Type":"ContainerDied","Data":"f9aedbe2d7ce81b64875e36b7db86ae3758c840b54e28ea48a91cbb44073818b"}
Nov 03 22:04:15.744871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.742282    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"
Nov 03 22:04:15.748007 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.747979    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\") pod \"cbc95a86-3a17-4287-afad-4b0dbdf44282\" (UID: \"cbc95a86-3a17-4287-afad-4b0dbdf44282\") "
Nov 03 22:04:15.748260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748235    5533 scope.go:122] "RemoveContainer" containerID="f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"
Nov 03 22:04:15.748502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748484    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748509    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748526    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc95a86-3a17-4287-afad-4b0dbdf44282-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748541    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748555    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748568    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748583    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x9r\" (UniqueName: \"kubernetes.io/projected/651af5cf-fbb4-4ade-b468-d1de7b667163-kube-api-access-74x9r\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748596    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748609    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.748608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748621    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748671    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7g9\" (UniqueName: \"kubernetes.io/projected/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-kube-api-access-9w7g9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748686    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2xz\" (UniqueName: \"kubernetes.io/projected/cbc95a86-3a17-4287-afad-4b0dbdf44282-kube-api-access-tc2xz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748702    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651af5cf-fbb4-4ade-b468-d1de7b667163-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748718    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748731    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7df02540-271e-4f6e-bc1d-1778fc2f5f0e-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748745    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748759    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651af5cf-fbb4-4ade-b468-d1de7b667163-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.748967    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache" pod "cbc95a86-3a17-4287-afad-4b0dbdf44282" (UID: "cbc95a86-3a17-4287-afad-4b0dbdf44282"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:15.749145    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787\": container with ID starting with f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787 not found: ID does not exist" containerID="f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749180    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787"} err="failed to get container status \"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787\": rpc error: code = NotFound desc = could not find container \"f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787\": container with ID starting with f7c75fdddbdeb9a0d0a903c2a37c8319edfe705381f2e52e316f17202fcee787 not found: ID does not exist"
Nov 03 22:04:15.749387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749202    5533 scope.go:122] "RemoveContainer" containerID="08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"
Nov 03 22:04:15.750248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749653    5533 generic.go:356] "Generic (PLEG): container finished" podID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerID="174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911" exitCode=0
Nov 03 22:04:15.750248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749707    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" event={"ID":"cbc95a86-3a17-4287-afad-4b0dbdf44282","Type":"ContainerDied","Data":"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"}
Nov 03 22:04:15.750248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749734    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj" event={"ID":"cbc95a86-3a17-4287-afad-4b0dbdf44282","Type":"ContainerDied","Data":"1855a767ad0f1a6bcd9b5514652886c3958a609f231c9b77ee5edd7b7fbcb2de"}
Nov 03 22:04:15.750248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.749958    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"
Nov 03 22:04:15.766806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.766757    5533 scope.go:122] "RemoveContainer" containerID="08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"
Nov 03 22:04:15.767978 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:15.767944    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f\": container with ID starting with 08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f not found: ID does not exist" containerID="08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"
Nov 03 22:04:15.768068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.767977    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f"} err="failed to get container status \"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f\": rpc error: code = NotFound desc = could not find container \"08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f\": container with ID starting with 08faa66c1fd98cd2dc78c3c7aefd1b199a991fceec4670a0896bf9ed2c7f8e8f not found: ID does not exist"
Nov 03 22:04:15.768068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.767999    5533 scope.go:122] "RemoveContainer" containerID="174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"
Nov 03 22:04:15.783480 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.783447    5533 scope.go:122] "RemoveContainer" containerID="174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"
Nov 03 22:04:15.783822 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:15.783793    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911\": container with ID starting with 174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911 not found: ID does not exist" containerID="174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"
Nov 03 22:04:15.783906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.783824    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911"} err="failed to get container status \"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911\": rpc error: code = NotFound desc = could not find container \"174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911\": container with ID starting with 174f8a8cec720d9572e4c9ea05193724c4d0aa6a2a95666b39fd13a5cf819911 not found: ID does not exist"
Nov 03 22:04:15.849957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:15.849920    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbc95a86-3a17-4287-afad-4b0dbdf44282-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:16.035715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.035679    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 22:04:16.040043 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.040016    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-2-w3-320f60-kserve-77ddd6d956-2gkbm"]
Nov 03 22:04:16.142766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.142695    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 22:04:16.146886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.146859    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/premium-w3-320f60-kserve-5d8656877b-nv7rj"]
Nov 03 22:04:16.249224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.249195    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 22:04:16.253884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.253857    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8"]
Nov 03 22:04:16.493683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.493656    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" path="/var/lib/kubelet/pods/651af5cf-fbb4-4ade-b468-d1de7b667163/volumes"
Nov 03 22:04:16.494141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.494119    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" path="/var/lib/kubelet/pods/7df02540-271e-4f6e-bc1d-1778fc2f5f0e/volumes"
Nov 03 22:04:16.494670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.494623    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" path="/var/lib/kubelet/pods/cbc95a86-3a17-4287-afad-4b0dbdf44282/volumes"
Nov 03 22:04:16.615939 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.615902    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-320f60/distinct-w3-320f60-kserve-58bd67b8c-bnpg8" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Nov 03 22:04:16.759682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.759586    5533 generic.go:356] "Generic (PLEG): container finished" podID="3783ee3b-5f53-4705-b690-20030433d97f" containerID="d44f6670763dfe21fe1f69dccfd3a3af0da90185119475957e36d9f683943d84" exitCode=0
Nov 03 22:04:16.760087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:16.759676    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk" event={"ID":"3783ee3b-5f53-4705-b690-20030433d97f","Type":"ContainerDied","Data":"d44f6670763dfe21fe1f69dccfd3a3af0da90185119475957e36d9f683943d84"}
Nov 03 22:04:17.907554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.907529    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:17.968175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.968152    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\") pod \"3783ee3b-5f53-4705-b690-20030433d97f\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") "
Nov 03 22:04:17.968314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.968187    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\") pod \"3783ee3b-5f53-4705-b690-20030433d97f\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") "
Nov 03 22:04:17.968314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.968294    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\") pod \"3783ee3b-5f53-4705-b690-20030433d97f\" (UID: \"3783ee3b-5f53-4705-b690-20030433d97f\") "
Nov 03 22:04:17.968693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.968664    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca" pod "3783ee3b-5f53-4705-b690-20030433d97f" (UID: "3783ee3b-5f53-4705-b690-20030433d97f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:17.970128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.970095    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca" pod "3783ee3b-5f53-4705-b690-20030433d97f" (UID: "3783ee3b-5f53-4705-b690-20030433d97f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:04:17.970286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:17.970260    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l" pod "3783ee3b-5f53-4705-b690-20030433d97f" (UID: "3783ee3b-5f53-4705-b690-20030433d97f"). InnerVolumeSpecName "kube-api-access-4nv7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:18.069610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.069548    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:18.069610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.069575    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv7l\" (UniqueName: \"kubernetes.io/projected/3783ee3b-5f53-4705-b690-20030433d97f-kube-api-access-4nv7l\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:18.069610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.069586    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3783ee3b-5f53-4705-b690-20030433d97f-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:18.778038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.777967    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"
Nov 03 22:04:18.778038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.777991    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk" event={"ID":"3783ee3b-5f53-4705-b690-20030433d97f","Type":"ContainerDied","Data":"304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932"}
Nov 03 22:04:18.778211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:18.778068    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304fdbb9fdec72d6b6f67809a9ae16b54ca35ab376fadc99d59340849a52d932"
Nov 03 22:04:19.262040 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:19.261996    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:04:19.994755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:19.994719    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 22:04:21.856022 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:21.855981    5533 scope.go:122] "RemoveContainer" containerID="dcc9bec5cbac01e09424a7ccec82b9eafbda9003733fe4e4a661c27e146fbf2b"
Nov 03 22:04:24.995753 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:24.995694    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" podUID="185d0df5-0fb4-4686-bf57-1a792d10d203" containerName="payload-processing" containerID="cri-o://c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce" gracePeriod=25
Nov 03 22:04:25.249035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.248979    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 22:04:25.342484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.342456    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\" (UniqueName: \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\") pod \"185d0df5-0fb4-4686-bf57-1a792d10d203\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") "
Nov 03 22:04:25.342629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.342602    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\") pod \"185d0df5-0fb4-4686-bf57-1a792d10d203\" (UID: \"185d0df5-0fb4-4686-bf57-1a792d10d203\") "
Nov 03 22:04:25.343021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.343000    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume" pod "185d0df5-0fb4-4686-bf57-1a792d10d203" (UID: "185d0df5-0fb4-4686-bf57-1a792d10d203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:25.344439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.344416    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh" pod "185d0df5-0fb4-4686-bf57-1a792d10d203" (UID: "185d0df5-0fb4-4686-bf57-1a792d10d203"). InnerVolumeSpecName "kube-api-access-pxddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:25.444117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.444088    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxddh\" (UniqueName: \"kubernetes.io/projected/185d0df5-0fb4-4686-bf57-1a792d10d203-kube-api-access-pxddh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:25.444117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.444118    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/185d0df5-0fb4-4686-bf57-1a792d10d203-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:25.829548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.829517    5533 generic.go:356] "Generic (PLEG): container finished" podID="185d0df5-0fb4-4686-bf57-1a792d10d203" containerID="c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce" exitCode=0
Nov 03 22:04:25.829756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.829584    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"
Nov 03 22:04:25.829756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.829621    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" event={"ID":"185d0df5-0fb4-4686-bf57-1a792d10d203","Type":"ContainerDied","Data":"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"}
Nov 03 22:04:25.829756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.829695    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq" event={"ID":"185d0df5-0fb4-4686-bf57-1a792d10d203","Type":"ContainerDied","Data":"473486ff6c09ec29d65437b4dc340a3ed55dc2326c10674a80e1e3a2f565ab43"}
Nov 03 22:04:25.829960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.829762    5533 scope.go:122] "RemoveContainer" containerID="c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"
Nov 03 22:04:25.840164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.840149    5533 scope.go:122] "RemoveContainer" containerID="c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"
Nov 03 22:04:25.840452 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:25.840432    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce\": container with ID starting with c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce not found: ID does not exist" containerID="c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"
Nov 03 22:04:25.840516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.840461    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce"} err="failed to get container status \"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce\": rpc error: code = NotFound desc = could not find container \"c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce\": container with ID starting with c25d5a169e5b217dad076efa02bf719cc3c07fb4dee3726fd94cc01d0010e3ce not found: ID does not exist"
Nov 03 22:04:25.985319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:25.985282    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 22:04:26.031646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:26.031600    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-320f60-6458c8fdbb-r7fjq"]
Nov 03 22:04:26.495859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:26.495823    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185d0df5-0fb4-4686-bf57-1a792d10d203" path="/var/lib/kubelet/pods/185d0df5-0fb4-4686-bf57-1a792d10d203/volumes"
Nov 03 22:04:44.266020 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:44.265979    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 22:04:44.266571 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:44.266257    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy" containerID="cri-o://a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6" gracePeriod=30
Nov 03 22:04:49.262231 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:49.262194    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:04:49.526121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.526068    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 22:04:49.691815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691773    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.691990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691823    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.691990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691855    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.691990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691887    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.691990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691918    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.691990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.691986    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.692280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.692055    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.692280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.692125    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.692280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.692166    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\") pod \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\" (UID: \"f6c65baa-882c-4cce-b5c7-5754d1fbff81\") "
Nov 03 22:04:49.692813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.692783    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:49.693343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.693253    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:49.693343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.693318    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:49.693543 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.693345    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:49.694190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.694152    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:49.695619 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.695582    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:04:49.696232 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.696202    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:49.696341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.696239    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "kube-api-access-7cgrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:49.702993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.702967    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy" pod "f6c65baa-882c-4cce-b5c7-5754d1fbff81" (UID: "f6c65baa-882c-4cce-b5c7-5754d1fbff81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793878    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793900    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793910    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793919    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793928    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.793940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793937    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.794339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793946    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cgrw\" (UniqueName: \"kubernetes.io/projected/f6c65baa-882c-4cce-b5c7-5754d1fbff81-kube-api-access-7cgrw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.794339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793955    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c65baa-882c-4cce-b5c7-5754d1fbff81-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.794339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.793964    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c65baa-882c-4cce-b5c7-5754d1fbff81-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:04:49.996133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.996092    5533 generic.go:356] "Generic (PLEG): container finished" podID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerID="a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6" exitCode=0
Nov 03 22:04:49.996302 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.996140    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"
Nov 03 22:04:49.996302 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.996141    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" event={"ID":"f6c65baa-882c-4cce-b5c7-5754d1fbff81","Type":"ContainerDied","Data":"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"}
Nov 03 22:04:49.996302 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.996202    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc" event={"ID":"f6c65baa-882c-4cce-b5c7-5754d1fbff81","Type":"ContainerDied","Data":"16a6950fe06151e733d6829493ff52d3ab3ceb393f5e5e16d0bbbadb5c6348f4"}
Nov 03 22:04:49.996302 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:49.996226    5533 scope.go:122] "RemoveContainer" containerID="a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"
Nov 03 22:04:50.007594 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:50.007573    5533 scope.go:122] "RemoveContainer" containerID="a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"
Nov 03 22:04:50.007929 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:04:50.007908    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6\": container with ID starting with a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6 not found: ID does not exist" containerID="a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"
Nov 03 22:04:50.007988 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:50.007938    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6"} err="failed to get container status \"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6\": rpc error: code = NotFound desc = could not find container \"a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6\": container with ID starting with a237e74779680d7110356a6d243dde9b1d34e3abbd74a5806919280790439ba6 not found: ID does not exist"
Nov 03 22:04:50.152730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:50.152018    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 22:04:50.155747 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:50.155715    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-320f60-openshift-default-768476b577-jzgrc"]
Nov 03 22:04:50.493763 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:50.493721    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" path="/var/lib/kubelet/pods/f6c65baa-882c-4cce-b5c7-5754d1fbff81/volumes"
Nov 03 22:04:55.574273 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:04:55.574242    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:05:00.354502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354423    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"]
Nov 03 22:05:00.354955 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354941    5533 cpu_manager.go:414] "containerMap: removing container" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354956    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354970    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354975    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354983    5533 cpu_manager.go:414] "containerMap: removing container" podUID="185d0df5-0fb4-4686-bf57-1a792d10d203" containerName="payload-processing"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.354988    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185d0df5-0fb4-4686-bf57-1a792d10d203" containerName="payload-processing"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355005    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355010    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main"
Nov 03 22:05:00.355016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355017    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355023    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355032    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3783ee3b-5f53-4705-b690-20030433d97f" containerName="revoke-keys"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355036    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3783ee3b-5f53-4705-b690-20030433d97f" containerName="revoke-keys"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355124    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="651af5cf-fbb4-4ade-b468-d1de7b667163" containerName="main"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355133    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c65baa-882c-4cce-b5c7-5754d1fbff81" containerName="istio-proxy"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355138    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="185d0df5-0fb4-4686-bf57-1a792d10d203" containerName="payload-processing"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355148    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df02540-271e-4f6e-bc1d-1778fc2f5f0e" containerName="main"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355153    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc95a86-3a17-4287-afad-4b0dbdf44282" containerName="main"
Nov 03 22:05:00.355281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.355159    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3783ee3b-5f53-4705-b690-20030433d97f" containerName="revoke-keys"
Nov 03 22:05:00.358745 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.358724    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.384258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.384236    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.384381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.384275    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rrl\" (UniqueName: \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.384436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.384385    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.408544 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.408517    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:05:00.454865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.454845    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:05:00.457513 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.457494    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"]
Nov 03 22:05:00.485865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.485839    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.485994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.485879    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.485994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.485922    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rrl\" (UniqueName: \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.486536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.486519    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.488140 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.488119    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.543972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.543951    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rrl\" (UniqueName: \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\") pod \"maas-api-revoke-sub-16d2bf94-gzk26\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:00.672010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:00.671989    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:01.040234 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:01.040207    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"]
Nov 03 22:05:01.042753 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:05:01.042719    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1247b40b_55ca_432a_83a5_f66793bc6029.slice/crio-8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897 WatchSource:0}: Error finding container 8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897: Status 404 returned error can't find the container with id 8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897
Nov 03 22:05:01.070971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:01.070931    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26" event={"ID":"1247b40b-55ca-432a-83a5-f66793bc6029","Type":"ContainerStarted","Data":"8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897"}
Nov 03 22:05:02.082536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:02.082502    5533 generic.go:356] "Generic (PLEG): container finished" podID="1247b40b-55ca-432a-83a5-f66793bc6029" containerID="b585a74243e46f5fae89d6108ca03b9e9e58633562fe2ef5653bc9e16a9db42f" exitCode=0
Nov 03 22:05:02.082908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:02.082583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26" event={"ID":"1247b40b-55ca-432a-83a5-f66793bc6029","Type":"ContainerDied","Data":"b585a74243e46f5fae89d6108ca03b9e9e58633562fe2ef5653bc9e16a9db42f"}
Nov 03 22:05:03.231473 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.231448    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:03.312147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.312122    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\") pod \"1247b40b-55ca-432a-83a5-f66793bc6029\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") "
Nov 03 22:05:03.312294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.312163    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\") pod \"1247b40b-55ca-432a-83a5-f66793bc6029\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") "
Nov 03 22:05:03.312294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.312183    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\" (UniqueName: \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\") pod \"1247b40b-55ca-432a-83a5-f66793bc6029\" (UID: \"1247b40b-55ca-432a-83a5-f66793bc6029\") "
Nov 03 22:05:03.312486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.312457    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca" pod "1247b40b-55ca-432a-83a5-f66793bc6029" (UID: "1247b40b-55ca-432a-83a5-f66793bc6029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:03.312568 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.312545    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:03.314209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.314190    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca" pod "1247b40b-55ca-432a-83a5-f66793bc6029" (UID: "1247b40b-55ca-432a-83a5-f66793bc6029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:03.314333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.314314    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl" pod "1247b40b-55ca-432a-83a5-f66793bc6029" (UID: "1247b40b-55ca-432a-83a5-f66793bc6029"). InnerVolumeSpecName "kube-api-access-h5rrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:03.413206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.413130    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1247b40b-55ca-432a-83a5-f66793bc6029-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:03.413206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:03.413159    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5rrl\" (UniqueName: \"kubernetes.io/projected/1247b40b-55ca-432a-83a5-f66793bc6029-kube-api-access-h5rrl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:04.105597 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:04.105550    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26" event={"ID":"1247b40b-55ca-432a-83a5-f66793bc6029","Type":"ContainerDied","Data":"8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897"}
Nov 03 22:05:04.105807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:04.105616    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"
Nov 03 22:05:04.105807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:04.105651    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eac474200e7d7703315ddea03da4a4359f57d407cec9f1858120ae4932d4897"
Nov 03 22:05:10.330611 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:10.330578    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 22:05:10.331087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:10.330813    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" containerID="cri-o://d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6" gracePeriod=30
Nov 03 22:05:11.070444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.070409    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"]
Nov 03 22:05:11.070941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.070925    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1247b40b-55ca-432a-83a5-f66793bc6029" containerName="revoke-keys"
Nov 03 22:05:11.071010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.070943    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1247b40b-55ca-432a-83a5-f66793bc6029" containerName="revoke-keys"
Nov 03 22:05:11.071045 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.071036    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1247b40b-55ca-432a-83a5-f66793bc6029" containerName="revoke-keys"
Nov 03 22:05:11.074479 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.074460    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.125653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.125615    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:05:11.129319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.129293    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"]
Nov 03 22:05:11.172300 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.172272    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:05:11.189925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.189904    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.190035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.189953    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.190140 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.190117    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.291243 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.291215    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.291349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.291266    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.291349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.291310    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.292077 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.292057    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.293781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.293759    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.348686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.348611    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\") pod \"maas-api-revoke-sub-76ad99dc-85598\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.371351 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.371327    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"]
Nov 03 22:05:11.375687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.375665    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.387796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.387777    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:11.431529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.431493    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"]
Nov 03 22:05:11.493759 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.493725    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2c8j\" (UniqueName: \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.493900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.493830    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.493948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.493899    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.595173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.595139    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.595360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.595306    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c8j\" (UniqueName: \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.595435 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.595359    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.596215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.596182    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.597618 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.597595    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.653617 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.653542    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c8j\" (UniqueName: \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\") pod \"maas-api-revoke-sub-c413af70-k5t6h\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.689289 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.689263    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:11.769804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:11.769770    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"]
Nov 03 22:05:11.771211 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:05:11.771182    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4808ad_6486_4660_bd87_961f0deda0cd.slice/crio-72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba WatchSource:0}: Error finding container 72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba: Status 404 returned error can't find the container with id 72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba
Nov 03 22:05:12.058513 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:12.058486    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"]
Nov 03 22:05:12.059407 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:05:12.059381    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9f51b6_7f22_453b_a8b3_62648e80ecc9.slice/crio-34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1 WatchSource:0}: Error finding container 34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1: Status 404 returned error can't find the container with id 34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1
Nov 03 22:05:12.159336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:12.159305    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598" event={"ID":"0e4808ad-6486-4660-bd87-961f0deda0cd","Type":"ContainerStarted","Data":"0bf2fcd36bad1ba071c11a507740e5b7df9ce352f8495b7c796c3c72564ceb4c"}
Nov 03 22:05:12.159460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:12.159361    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598" event={"ID":"0e4808ad-6486-4660-bd87-961f0deda0cd","Type":"ContainerStarted","Data":"72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba"}
Nov 03 22:05:12.160630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:12.160606    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h" event={"ID":"fe9f51b6-7f22-453b-a8b3-62648e80ecc9","Type":"ContainerStarted","Data":"34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1"}
Nov 03 22:05:12.268936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:12.268894    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598" podStartSLOduration=1.268881907 podStartE2EDuration="1.268881907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:12.267611338 +0000 UTC m=+4613.898924815" watchObservedRunningTime="2026-11-03 22:05:12.268881907 +0000 UTC m=+4613.900195400"
Nov 03 22:05:13.172967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.172934    5533 generic.go:356] "Generic (PLEG): container finished" podID="0e4808ad-6486-4660-bd87-961f0deda0cd" containerID="0bf2fcd36bad1ba071c11a507740e5b7df9ce352f8495b7c796c3c72564ceb4c" exitCode=0
Nov 03 22:05:13.173460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.173029    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598" event={"ID":"0e4808ad-6486-4660-bd87-961f0deda0cd","Type":"ContainerDied","Data":"0bf2fcd36bad1ba071c11a507740e5b7df9ce352f8495b7c796c3c72564ceb4c"}
Nov 03 22:05:13.174775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.174751    5533 generic.go:356] "Generic (PLEG): container finished" podID="fe9f51b6-7f22-453b-a8b3-62648e80ecc9" containerID="3a41f2564d6ad256e975f3dd62831f17c3f5e51df599cdd89b13b4c04247e9c8" exitCode=0
Nov 03 22:05:13.174891 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.174806    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h" event={"ID":"fe9f51b6-7f22-453b-a8b3-62648e80ecc9","Type":"ContainerDied","Data":"3a41f2564d6ad256e975f3dd62831f17c3f5e51df599cdd89b13b4c04247e9c8"}
Nov 03 22:05:13.592871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.592832    5533 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:05:13.593047 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:13.592893    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:05:14.358171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.358150    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:14.362313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.362293    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:14.421597 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421571    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\") pod \"0e4808ad-6486-4660-bd87-961f0deda0cd\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") "
Nov 03 22:05:14.421705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421664    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\") pod \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") "
Nov 03 22:05:14.421750 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421732    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\") pod \"0e4808ad-6486-4660-bd87-961f0deda0cd\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") "
Nov 03 22:05:14.421823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421808    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\") pod \"0e4808ad-6486-4660-bd87-961f0deda0cd\" (UID: \"0e4808ad-6486-4660-bd87-961f0deda0cd\") "
Nov 03 22:05:14.421912 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421896    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\") pod \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") "
Nov 03 22:05:14.422006 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.421981    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\" (UniqueName: \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\") pod \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\" (UID: \"fe9f51b6-7f22-453b-a8b3-62648e80ecc9\") "
Nov 03 22:05:14.422220 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.422109    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca" pod "0e4808ad-6486-4660-bd87-961f0deda0cd" (UID: "0e4808ad-6486-4660-bd87-961f0deda0cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:14.422356 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.422323    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca" pod "fe9f51b6-7f22-453b-a8b3-62648e80ecc9" (UID: "fe9f51b6-7f22-453b-a8b3-62648e80ecc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:14.422490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.422397    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:14.423952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.423923    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6" pod "0e4808ad-6486-4660-bd87-961f0deda0cd" (UID: "0e4808ad-6486-4660-bd87-961f0deda0cd"). InnerVolumeSpecName "kube-api-access-5ggr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:14.424219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.424197    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca" pod "fe9f51b6-7f22-453b-a8b3-62648e80ecc9" (UID: "fe9f51b6-7f22-453b-a8b3-62648e80ecc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:14.424339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.424320    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j" pod "fe9f51b6-7f22-453b-a8b3-62648e80ecc9" (UID: "fe9f51b6-7f22-453b-a8b3-62648e80ecc9"). InnerVolumeSpecName "kube-api-access-j2c8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:14.424482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.424460    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca" pod "0e4808ad-6486-4660-bd87-961f0deda0cd" (UID: "0e4808ad-6486-4660-bd87-961f0deda0cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:14.523912 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.523884    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4808ad-6486-4660-bd87-961f0deda0cd-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:14.523912 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.523911    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:14.524052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.523921    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2c8j\" (UniqueName: \"kubernetes.io/projected/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-kube-api-access-j2c8j\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:14.524052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.523931    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/0e4808ad-6486-4660-bd87-961f0deda0cd-kube-api-access-5ggr6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:14.524052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:14.523941    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe9f51b6-7f22-453b-a8b3-62648e80ecc9-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.188889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.188854    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"
Nov 03 22:05:15.189093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.188885    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598" event={"ID":"0e4808ad-6486-4660-bd87-961f0deda0cd","Type":"ContainerDied","Data":"72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba"}
Nov 03 22:05:15.189093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.188946    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72ffc228f2a01ed9096e05a151566c65df7acb8be38a73706107307698b539ba"
Nov 03 22:05:15.190369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.190346    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h" event={"ID":"fe9f51b6-7f22-453b-a8b3-62648e80ecc9","Type":"ContainerDied","Data":"34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1"}
Nov 03 22:05:15.190476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.190380    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e5175330a61ad75ad470770d2ca84dbacd6b9c6eca35653d2244f7033951f1"
Nov 03 22:05:15.190476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.190381    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"
Nov 03 22:05:15.603004 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.602979    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 22:05:15.734522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734450    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734489    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734546    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734620    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734691    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734726    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.734780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734757    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.735070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734857    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.735070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.734896    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\") pod \"c2502850-affd-4c72-8e42-adce28381d06\" (UID: \"c2502850-affd-4c72-8e42-adce28381d06\") "
Nov 03 22:05:15.735377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.735061    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:15.735477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.735072    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:15.735477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.735126    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:15.735477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.735232    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:15.735477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.735307    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:15.736827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.736794    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:05:15.736939 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.736873    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:15.737450 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.737427    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "kube-api-access-42ggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:15.737497 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.737461    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token" pod "c2502850-affd-4c72-8e42-adce28381d06" (UID: "c2502850-affd-4c72-8e42-adce28381d06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:15.775761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.775734    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 22:05:15.836997 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.836967    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2502850-affd-4c72-8e42-adce28381d06-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837000    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837017    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837030    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837044    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ggf\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-kube-api-access-42ggf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837059    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837072    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2502850-affd-4c72-8e42-adce28381d06-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837085    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2502850-affd-4c72-8e42-adce28381d06-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:15.837106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:15.837099    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2502850-affd-4c72-8e42-adce28381d06-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:16.201377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.201345    5533 generic.go:356] "Generic (PLEG): container finished" podID="c2502850-affd-4c72-8e42-adce28381d06" containerID="d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6" exitCode=0
Nov 03 22:05:16.201587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.201392    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"
Nov 03 22:05:16.201587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.201423    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerDied","Data":"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"}
Nov 03 22:05:16.201587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.201490    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r" event={"ID":"c2502850-affd-4c72-8e42-adce28381d06","Type":"ContainerDied","Data":"0d3414dd9d160ce64a73ba14dd658537a972bf522c72ae8abe34c42e03633490"}
Nov 03 22:05:16.201587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.201521    5533 scope.go:122] "RemoveContainer" containerID="d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"
Nov 03 22:05:16.212696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.212681    5533 scope.go:122] "RemoveContainer" containerID="8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"
Nov 03 22:05:16.221566 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.221550    5533 scope.go:122] "RemoveContainer" containerID="d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"
Nov 03 22:05:16.221833 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:16.221811    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6\": container with ID starting with d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6 not found: ID does not exist" containerID="d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"
Nov 03 22:05:16.221893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.221841    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6"} err="failed to get container status \"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6\": rpc error: code = NotFound desc = could not find container \"d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6\": container with ID starting with d02e701b35a1303b0388c8a4e8e8c8da7010a728fbd59d6c076bf34088695fd6 not found: ID does not exist"
Nov 03 22:05:16.221893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.221859    5533 scope.go:122] "RemoveContainer" containerID="8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"
Nov 03 22:05:16.222106 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:16.222088    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708\": container with ID starting with 8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708 not found: ID does not exist" containerID="8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"
Nov 03 22:05:16.222154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.222112    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708"} err="failed to get container status \"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708\": rpc error: code = NotFound desc = could not find container \"8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708\": container with ID starting with 8efedc86ab79b9a01148be0f9dea5a7419a7bfe842825ec36a5a8f66e848e708 not found: ID does not exist"
Nov 03 22:05:16.367945 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.367922    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 22:05:16.374126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.374103    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0dc66-openshift-default-5946957f9-n6f7r"]
Nov 03 22:05:16.493605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:16.493548    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2502850-affd-4c72-8e42-adce28381d06" path="/var/lib/kubelet/pods/c2502850-affd-4c72-8e42-adce28381d06/volumes"
Nov 03 22:05:19.261346 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:19.261305    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:05:24.180387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.180353    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"]
Nov 03 22:05:24.181137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181106    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0e4808ad-6486-4660-bd87-961f0deda0cd" containerName="revoke-keys"
Nov 03 22:05:24.181137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181132    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4808ad-6486-4660-bd87-961f0deda0cd" containerName="revoke-keys"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181149    5533 cpu_manager.go:414] "containerMap: removing container" podUID="fe9f51b6-7f22-453b-a8b3-62648e80ecc9" containerName="revoke-keys"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181157    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9f51b6-7f22-453b-a8b3-62648e80ecc9" containerName="revoke-keys"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181175    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181183    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181204    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181211    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.181330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181316    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.181730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181338    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4808ad-6486-4660-bd87-961f0deda0cd" containerName="revoke-keys"
Nov 03 22:05:24.181730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181350    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9f51b6-7f22-453b-a8b3-62648e80ecc9" containerName="revoke-keys"
Nov 03 22:05:24.181730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.181360    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2502850-affd-4c72-8e42-adce28381d06" containerName="istio-proxy"
Nov 03 22:05:24.186698 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.186676    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.236355 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.236326    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:05:24.238558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.238533    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"]
Nov 03 22:05:24.281323 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.281296    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:05:24.318130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.318105    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfb8z\" (UniqueName: \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.318259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.318239    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.318313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.318286    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.419492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.419467    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.419599 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.419505    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.419599 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.419541    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfb8z\" (UniqueName: \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.420233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.420204    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.421935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.421913    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.477493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.477433    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfb8z\" (UniqueName: \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\") pod \"maas-api-revoke-sub-e0732a06-wlct4\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.504323 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.504299    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:24.873100 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:24.873074    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"]
Nov 03 22:05:24.874867 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:05:24.874836    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b062cd7_d2c6_44ad_9199_d1b2c18d2cd2.slice/crio-61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf WatchSource:0}: Error finding container 61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf: Status 404 returned error can't find the container with id 61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf
Nov 03 22:05:25.265593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:25.265556    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4" event={"ID":"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2","Type":"ContainerStarted","Data":"6c1d0a9853ba003f31edc62f7299c4d3f6cbee85d2e900143042a521aa510466"}
Nov 03 22:05:25.265974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:25.265622    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4" event={"ID":"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2","Type":"ContainerStarted","Data":"61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf"}
Nov 03 22:05:25.376958 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:25.376914    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4" podStartSLOduration=1.3769012950000001 podStartE2EDuration="1.376901295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:25.373851347 +0000 UTC m=+4627.005164825" watchObservedRunningTime="2026-11-03 22:05:25.376901295 +0000 UTC m=+4627.008214772"
Nov 03 22:05:26.274872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:26.274832    5533 generic.go:356] "Generic (PLEG): container finished" podID="9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" containerID="6c1d0a9853ba003f31edc62f7299c4d3f6cbee85d2e900143042a521aa510466" exitCode=0
Nov 03 22:05:26.275269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:26.274880    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4" event={"ID":"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2","Type":"ContainerDied","Data":"6c1d0a9853ba003f31edc62f7299c4d3f6cbee85d2e900143042a521aa510466"}
Nov 03 22:05:26.439349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:26.439319    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 22:05:26.548552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:26.548456    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 22:05:27.425034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.425007    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:27.548759 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.548672    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\") pod \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") "
Nov 03 22:05:27.548926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.548760    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\") pod \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") "
Nov 03 22:05:27.548926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.548804    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\" (UniqueName: \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\") pod \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\" (UID: \"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2\") "
Nov 03 22:05:27.549179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.549147    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca" pod "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" (UID: "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:27.551212 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.551173    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z" pod "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" (UID: "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2"). InnerVolumeSpecName "kube-api-access-dfb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:27.553589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.553565    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca" pod "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" (UID: "9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:27.649560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.649529    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:27.649560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.649553    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:27.649560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:27.649564    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfb8z\" (UniqueName: \"kubernetes.io/projected/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2-kube-api-access-dfb8z\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:28.291260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:28.291219    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"
Nov 03 22:05:28.291452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:28.291263    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4" event={"ID":"9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2","Type":"ContainerDied","Data":"61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf"}
Nov 03 22:05:28.291452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:28.291306    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61069a0c474ac5fba6c553fa813784687491cf3ae3d864b02252d1b9cf3ebecf"
Nov 03 22:05:30.795487 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:30.795446    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main" containerID="cri-o://7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb" gracePeriod=45
Nov 03 22:05:31.051845 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.051788    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 22:05:31.076499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076472    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076523    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076576    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\" (UniqueName: \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076732 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076709    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076754    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076779    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:31.076850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076783    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\") pod \"0a72c370-7a7f-4c86-a56f-8a662d457a80\" (UID: \"0a72c370-7a7f-4c86-a56f-8a662d457a80\") "
Nov 03 22:05:31.076850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076804    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:31.077027 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.076991    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:31.077175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.077160    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.077239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.077176    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.077239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.077184    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.078843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.078819    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "kube-api-access-7gxmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:31.078940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.078851    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:05:31.078940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.078885    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs" pod "0a72c370-7a7f-4c86-a56f-8a662d457a80" (UID: "0a72c370-7a7f-4c86-a56f-8a662d457a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:05:31.178810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.178782    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxmp\" (UniqueName: \"kubernetes.io/projected/0a72c370-7a7f-4c86-a56f-8a662d457a80-kube-api-access-7gxmp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.178810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.178810    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a72c370-7a7f-4c86-a56f-8a662d457a80-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.178980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.178819    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a72c370-7a7f-4c86-a56f-8a662d457a80-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:31.317786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.317703    5533 generic.go:356] "Generic (PLEG): container finished" podID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerID="7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb" exitCode=0
Nov 03 22:05:31.317786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.317755    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"
Nov 03 22:05:31.317973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.317788    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" event={"ID":"0a72c370-7a7f-4c86-a56f-8a662d457a80","Type":"ContainerDied","Data":"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"}
Nov 03 22:05:31.317973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.317832    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq" event={"ID":"0a72c370-7a7f-4c86-a56f-8a662d457a80","Type":"ContainerDied","Data":"0d1708ea2a753cdf6c7be4774f23860cb7e430555a4cc005799b8026b2260c31"}
Nov 03 22:05:31.317973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.317853    5533 scope.go:122] "RemoveContainer" containerID="7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"
Nov 03 22:05:31.336419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.336400    5533 scope.go:122] "RemoveContainer" containerID="7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"
Nov 03 22:05:31.336829 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:31.336802    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb\": container with ID starting with 7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb not found: ID does not exist" containerID="7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"
Nov 03 22:05:31.336947 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.336831    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb"} err="failed to get container status \"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb\": rpc error: code = NotFound desc = could not find container \"7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb\": container with ID starting with 7192c5c9e940e73c91b3ce17c91b6ff22ede04f46450d5dfeaafdfbc0b7d2bdb not found: ID does not exist"
Nov 03 22:05:31.440184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.440154    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing" containerID="cri-o://de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc" gracePeriod=25
Nov 03 22:05:31.481457 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.481430    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 22:05:31.489416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.489375    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-791c32/premium-w0-791c32-kserve-dbb6d88d5-tllrq"]
Nov 03 22:05:31.550342 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:31.550280    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing" containerID="cri-o://ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764" gracePeriod=25
Nov 03 22:05:32.492556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:32.492522    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" path="/var/lib/kubelet/pods/0a72c370-7a7f-4c86-a56f-8a662d457a80/volumes"
Nov 03 22:05:38.244515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:38.244482    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:05:38.245010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:38.244520    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:05:48.244852 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:48.244810    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:05:48.245316 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:48.244818    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:05:49.261600 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:49.261566    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:05:51.713759 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.713691    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 22:05:51.776573 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.776541    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\" (UniqueName: \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\") pod \"67b82980-14f3-4dc9-80ac-2d8f4111a153\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") "
Nov 03 22:05:51.776779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.776681    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\") pod \"67b82980-14f3-4dc9-80ac-2d8f4111a153\" (UID: \"67b82980-14f3-4dc9-80ac-2d8f4111a153\") "
Nov 03 22:05:51.777135 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.777080    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume" pod "67b82980-14f3-4dc9-80ac-2d8f4111a153" (UID: "67b82980-14f3-4dc9-80ac-2d8f4111a153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:51.779169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.779138    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82" pod "67b82980-14f3-4dc9-80ac-2d8f4111a153" (UID: "67b82980-14f3-4dc9-80ac-2d8f4111a153"). InnerVolumeSpecName "kube-api-access-jrs82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:51.801151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.801125    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 22:05:51.877774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.877743    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\" (UniqueName: \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\") pod \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") "
Nov 03 22:05:51.877973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.877905    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\") pod \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\" (UID: \"f497549a-ea48-45fa-86c9-81fb4e98fd5a\") "
Nov 03 22:05:51.878114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.878098    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b82980-14f3-4dc9-80ac-2d8f4111a153-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:51.878174 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.878117    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrs82\" (UniqueName: \"kubernetes.io/projected/67b82980-14f3-4dc9-80ac-2d8f4111a153-kube-api-access-jrs82\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:51.878292 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.878270    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume" pod "f497549a-ea48-45fa-86c9-81fb4e98fd5a" (UID: "f497549a-ea48-45fa-86c9-81fb4e98fd5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:05:51.879892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.879869    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls" pod "f497549a-ea48-45fa-86c9-81fb4e98fd5a" (UID: "f497549a-ea48-45fa-86c9-81fb4e98fd5a"). InnerVolumeSpecName "kube-api-access-wmwls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:05:51.979132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.979055    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497549a-ea48-45fa-86c9-81fb4e98fd5a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:51.979132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:51.979093    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwls\" (UniqueName: \"kubernetes.io/projected/f497549a-ea48-45fa-86c9-81fb4e98fd5a-kube-api-access-wmwls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:05:52.462921 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.462893    5533 generic.go:356] "Generic (PLEG): container finished" podID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerID="de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc" exitCode=0
Nov 03 22:05:52.463086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.462952    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"
Nov 03 22:05:52.463086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.462981    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" event={"ID":"67b82980-14f3-4dc9-80ac-2d8f4111a153","Type":"ContainerDied","Data":"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"}
Nov 03 22:05:52.463086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.463049    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4" event={"ID":"67b82980-14f3-4dc9-80ac-2d8f4111a153","Type":"ContainerDied","Data":"3f3501435e258cd362994536c377e058bd237ee57af98fe9ea20f6a38cac8c91"}
Nov 03 22:05:52.463086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.463086    5533 scope.go:122] "RemoveContainer" containerID="de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"
Nov 03 22:05:52.464604 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.464580    5533 generic.go:356] "Generic (PLEG): container finished" podID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerID="ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764" exitCode=0
Nov 03 22:05:52.464813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.464661    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"
Nov 03 22:05:52.464813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.464765    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" event={"ID":"f497549a-ea48-45fa-86c9-81fb4e98fd5a","Type":"ContainerDied","Data":"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"}
Nov 03 22:05:52.464813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.464807    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk" event={"ID":"f497549a-ea48-45fa-86c9-81fb4e98fd5a","Type":"ContainerDied","Data":"e718caddc393ee6a4a66331912920ee74c6d567d9f6af28783b4a6b4167f1f5f"}
Nov 03 22:05:52.472980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.472960    5533 scope.go:122] "RemoveContainer" containerID="de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"
Nov 03 22:05:52.473243 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:52.473220    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc\": container with ID starting with de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc not found: ID does not exist" containerID="de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"
Nov 03 22:05:52.473314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.473251    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc"} err="failed to get container status \"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc\": rpc error: code = NotFound desc = could not find container \"de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc\": container with ID starting with de91a4b6635e4c02bc64c2e8700969571f01177b786d47dc1f3396d898c8bebc not found: ID does not exist"
Nov 03 22:05:52.473314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.473272    5533 scope.go:122] "RemoveContainer" containerID="ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"
Nov 03 22:05:52.486474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.486450    5533 scope.go:122] "RemoveContainer" containerID="ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"
Nov 03 22:05:52.486730 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:05:52.486706    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764\": container with ID starting with ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764 not found: ID does not exist" containerID="ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"
Nov 03 22:05:52.486825 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.486740    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764"} err="failed to get container status \"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764\": rpc error: code = NotFound desc = could not find container \"ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764\": container with ID starting with ef6ec112791df9ee4337f03cab208df7a24ccc273105631d4575340d40f2e764 not found: ID does not exist"
Nov 03 22:05:52.619556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.619521    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 22:05:52.667913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.667881    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-791c32-97dbd5dcb-q5jm4"]
Nov 03 22:05:52.732220 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.732156    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 22:05:52.736354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:52.736329    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-791c32-5c8898c67-6sthk"]
Nov 03 22:05:54.493551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:54.493519    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" path="/var/lib/kubelet/pods/67b82980-14f3-4dc9-80ac-2d8f4111a153/volumes"
Nov 03 22:05:54.493987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:54.493971    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" path="/var/lib/kubelet/pods/f497549a-ea48-45fa-86c9-81fb4e98fd5a/volumes"
Nov 03 22:05:58.008817 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:05:58.008784    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:06:03.525524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.525487    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"]
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526010    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526024    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526033    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" containerName="revoke-keys"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526038    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" containerName="revoke-keys"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526062    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526068    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526076    5533 cpu_manager.go:414] "containerMap: removing container" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526080    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526155    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f497549a-ea48-45fa-86c9-81fb4e98fd5a" containerName="payload-pre-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526163    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b82980-14f3-4dc9-80ac-2d8f4111a153" containerName="payload-processing"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526173    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a72c370-7a7f-4c86-a56f-8a662d457a80" containerName="main"
Nov 03 22:06:03.528040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.526180    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" containerName="revoke-keys"
Nov 03 22:06:03.529489 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.529473    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.584453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.584424    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"]
Nov 03 22:06:03.594213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.594193    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:06:03.594335 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.594287    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:06:03.687905 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.687880    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.688046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.687921    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.688046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.687942    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22fr\" (UniqueName: \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.789444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.789379    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.789444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.789425    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.789622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.789459    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22fr\" (UniqueName: \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.790176 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.790148    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.791766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.791748    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:03.846622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:03.846594    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22fr\" (UniqueName: \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\") pod \"maas-api-revoke-sub-0a2a7c34-48mfs\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:04.143368 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:04.143282    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:04.525396 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:04.525361    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"]
Nov 03 22:06:04.526697 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:06:04.526668    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4082ee_7e09_4e86_b9a5_1660d1ca6b6e.slice/crio-d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7 WatchSource:0}: Error finding container d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7: Status 404 returned error can't find the container with id d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7
Nov 03 22:06:04.545194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:04.545161    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs" event={"ID":"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e","Type":"ContainerStarted","Data":"d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7"}
Nov 03 22:06:05.554205 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:05.554175    5533 generic.go:356] "Generic (PLEG): container finished" podID="8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" containerID="bd0bf182bb828d917c8c28e2cee82c503a3e9fd41b0977a036eb92fede65a324" exitCode=0
Nov 03 22:06:05.554552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:05.554237    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs" event={"ID":"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e","Type":"ContainerDied","Data":"bd0bf182bb828d917c8c28e2cee82c503a3e9fd41b0977a036eb92fede65a324"}
Nov 03 22:06:06.705910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.705883    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:06.822294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.822233    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\" (UniqueName: \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\") pod \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") "
Nov 03 22:06:06.822294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.822269    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\") pod \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") "
Nov 03 22:06:06.822504 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.822342    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\") pod \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\" (UID: \"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e\") "
Nov 03 22:06:06.822823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.822794    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca" pod "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" (UID: "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:06.825668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.824681    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr" pod "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" (UID: "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e"). InnerVolumeSpecName "kube-api-access-p22fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:06.825668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.824708    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca" pod "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" (UID: "8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:06:06.923202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.923178    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22fr\" (UniqueName: \"kubernetes.io/projected/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-kube-api-access-p22fr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:06.923202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.923200    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:06.923343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:06.923210    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:07.568228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:07.568197    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"
Nov 03 22:06:07.568228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:07.568216    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs" event={"ID":"8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e","Type":"ContainerDied","Data":"d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7"}
Nov 03 22:06:07.568491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:07.568257    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0918e1ced18325c05c83a1dc8d4c8d31e6be9aa3753def58bb951fec802c3c7"
Nov 03 22:06:14.188318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.188275    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"]
Nov 03 22:06:14.188868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.188806    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" containerName="revoke-keys"
Nov 03 22:06:14.188868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.188822    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" containerName="revoke-keys"
Nov 03 22:06:14.188963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.188940    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" containerName="revoke-keys"
Nov 03 22:06:14.193015 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.192991    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.288372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.288341    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.288523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.288395    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6htc\" (UniqueName: \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.288523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.288495    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.289700 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.289674    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:06:14.289700 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.289690    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:06:14.302798 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.302757    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"]
Nov 03 22:06:14.390132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.390095    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.390283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.390166    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6htc\" (UniqueName: \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.390283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.390196    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.390849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.390828    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.392719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.392689    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.448378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.448319    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6htc\" (UniqueName: \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\") pod \"maas-api-revoke-sub-304ba485-95ssm\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.506702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.506680    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:14.882577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:14.882544    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"]
Nov 03 22:06:14.884111 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:06:14.884083    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f61bb42_4c0a_4d95_92f7_687b8d2d6320.slice/crio-2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8 WatchSource:0}: Error finding container 2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8: Status 404 returned error can't find the container with id 2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8
Nov 03 22:06:15.621796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:15.621761    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm" event={"ID":"6f61bb42-4c0a-4d95-92f7-687b8d2d6320","Type":"ContainerStarted","Data":"d980eb92ada9fc0917046bb66f2ba85d6b72c3185390c73aea716b4a54347fdf"}
Nov 03 22:06:15.622253 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:15.621812    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm" event={"ID":"6f61bb42-4c0a-4d95-92f7-687b8d2d6320","Type":"ContainerStarted","Data":"2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8"}
Nov 03 22:06:15.737166 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:15.737115    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm" podStartSLOduration=1.73709888 podStartE2EDuration="1.73709888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:15.733666215 +0000 UTC m=+4677.364979692" watchObservedRunningTime="2026-11-03 22:06:15.73709888 +0000 UTC m=+4677.368412356"
Nov 03 22:06:16.550461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:16.550427    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 22:06:16.631552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:16.631519    5533 generic.go:356] "Generic (PLEG): container finished" podID="6f61bb42-4c0a-4d95-92f7-687b8d2d6320" containerID="d980eb92ada9fc0917046bb66f2ba85d6b72c3185390c73aea716b4a54347fdf" exitCode=0
Nov 03 22:06:16.631949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:16.631611    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm" event={"ID":"6f61bb42-4c0a-4d95-92f7-687b8d2d6320","Type":"ContainerDied","Data":"d980eb92ada9fc0917046bb66f2ba85d6b72c3185390c73aea716b4a54347fdf"}
Nov 03 22:06:16.673349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:16.673315    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 22:06:17.783439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.783414    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:17.823042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.823018    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\") pod \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") "
Nov 03 22:06:17.823169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.823128    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\" (UniqueName: \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\") pod \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") "
Nov 03 22:06:17.823221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.823206    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\") pod \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\" (UID: \"6f61bb42-4c0a-4d95-92f7-687b8d2d6320\") "
Nov 03 22:06:17.823515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.823490    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca" pod "6f61bb42-4c0a-4d95-92f7-687b8d2d6320" (UID: "6f61bb42-4c0a-4d95-92f7-687b8d2d6320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:17.825036 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.825014    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca" pod "6f61bb42-4c0a-4d95-92f7-687b8d2d6320" (UID: "6f61bb42-4c0a-4d95-92f7-687b8d2d6320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:06:17.825114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.825034    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc" pod "6f61bb42-4c0a-4d95-92f7-687b8d2d6320" (UID: "6f61bb42-4c0a-4d95-92f7-687b8d2d6320"). InnerVolumeSpecName "kube-api-access-f6htc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:17.924318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.924294    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:17.924318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.924317    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:17.924474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:17.924327    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6htc\" (UniqueName: \"kubernetes.io/projected/6f61bb42-4c0a-4d95-92f7-687b8d2d6320-kube-api-access-f6htc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:18.646818 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:18.646794    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"
Nov 03 22:06:18.646980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:18.646826    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm" event={"ID":"6f61bb42-4c0a-4d95-92f7-687b8d2d6320","Type":"ContainerDied","Data":"2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8"}
Nov 03 22:06:18.646980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:18.646867    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2104717f1f54ae15fda779ad81351b63e6a871bfa94ead12305c353fb9f922c8"
Nov 03 22:06:19.259968 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:06:19.259934    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:06:21.550850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:21.550808    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing" containerID="cri-o://dd23d53e62a4828bfe74569a177c7a64202fae5f523d295c9bb462ecf9ac73b6" gracePeriod=25
Nov 03 22:06:21.674428 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:21.674364    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing" containerID="cri-o://c8f8fcf2cb55db9e3df150e92b00ee1ba8d676562c8022f465934a10ed7982c8" gracePeriod=25
Nov 03 22:06:27.286591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:27.286553    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:06:27.286591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:27.286572    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:06:30.412068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:30.411990    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:06:37.202526 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:37.202493    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:06:37.287063 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:37.287029    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:06:37.287219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:37.287064    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:06:41.799141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.799112    5533 generic.go:356] "Generic (PLEG): container finished" podID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerID="c8f8fcf2cb55db9e3df150e92b00ee1ba8d676562c8022f465934a10ed7982c8" exitCode=0
Nov 03 22:06:41.799488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.799191    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" event={"ID":"f3558e41-3e6a-47c8-84a7-1b068ca3285d","Type":"ContainerDied","Data":"c8f8fcf2cb55db9e3df150e92b00ee1ba8d676562c8022f465934a10ed7982c8"}
Nov 03 22:06:41.800862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.800841    5533 generic.go:356] "Generic (PLEG): container finished" podID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerID="dd23d53e62a4828bfe74569a177c7a64202fae5f523d295c9bb462ecf9ac73b6" exitCode=0
Nov 03 22:06:41.800991 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.800895    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" event={"ID":"5b006e79-d578-41a5-92c1-de0e5cf3551e","Type":"ContainerDied","Data":"dd23d53e62a4828bfe74569a177c7a64202fae5f523d295c9bb462ecf9ac73b6"}
Nov 03 22:06:41.800991 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.800936    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888" event={"ID":"5b006e79-d578-41a5-92c1-de0e5cf3551e","Type":"ContainerDied","Data":"559fbfaf738421efccf3fea475fc0bc84e46c996596cbc8e7dfae26f553679e7"}
Nov 03 22:06:41.800991 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.800961    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559fbfaf738421efccf3fea475fc0bc84e46c996596cbc8e7dfae26f553679e7"
Nov 03 22:06:41.817380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.817330    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 22:06:41.949974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.949951    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 22:06:41.961094 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.961067    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\" (UniqueName: \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\") pod \"5b006e79-d578-41a5-92c1-de0e5cf3551e\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") "
Nov 03 22:06:41.961223 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.961139    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\") pod \"5b006e79-d578-41a5-92c1-de0e5cf3551e\" (UID: \"5b006e79-d578-41a5-92c1-de0e5cf3551e\") "
Nov 03 22:06:41.961585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.961558    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume" pod "5b006e79-d578-41a5-92c1-de0e5cf3551e" (UID: "5b006e79-d578-41a5-92c1-de0e5cf3551e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:41.963180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:41.963151    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn" pod "5b006e79-d578-41a5-92c1-de0e5cf3551e" (UID: "5b006e79-d578-41a5-92c1-de0e5cf3551e"). InnerVolumeSpecName "kube-api-access-22zkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:42.061939 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.061907    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\") pod \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") "
Nov 03 22:06:42.062099 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.062082    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\" (UniqueName: \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\") pod \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\" (UID: \"f3558e41-3e6a-47c8-84a7-1b068ca3285d\") "
Nov 03 22:06:42.062281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.062256    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume" pod "f3558e41-3e6a-47c8-84a7-1b068ca3285d" (UID: "f3558e41-3e6a-47c8-84a7-1b068ca3285d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:42.062379 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.062367    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3558e41-3e6a-47c8-84a7-1b068ca3285d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:42.062456 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.062380    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b006e79-d578-41a5-92c1-de0e5cf3551e-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:42.062456 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.062390    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zkn\" (UniqueName: \"kubernetes.io/projected/5b006e79-d578-41a5-92c1-de0e5cf3551e-kube-api-access-22zkn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:42.064356 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.064332    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh" pod "f3558e41-3e6a-47c8-84a7-1b068ca3285d" (UID: "f3558e41-3e6a-47c8-84a7-1b068ca3285d"). InnerVolumeSpecName "kube-api-access-cccfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:42.163922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.163829    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccfh\" (UniqueName: \"kubernetes.io/projected/f3558e41-3e6a-47c8-84a7-1b068ca3285d-kube-api-access-cccfh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:42.808756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.808677    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg" event={"ID":"f3558e41-3e6a-47c8-84a7-1b068ca3285d","Type":"ContainerDied","Data":"43668c99791701ebaa9d6cd82834e2d4e45d7cc09de508a9ee6a7f29ac73ba98"}
Nov 03 22:06:42.808756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.808705    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"
Nov 03 22:06:42.809188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.808759    5533 scope.go:122] "RemoveContainer" containerID="c8f8fcf2cb55db9e3df150e92b00ee1ba8d676562c8022f465934a10ed7982c8"
Nov 03 22:06:42.809188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.808941    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"
Nov 03 22:06:42.964013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.963972    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 22:06:42.967978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:42.967950    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-4d1799-57b44467db-7s888"]
Nov 03 22:06:43.070111 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.070043    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 22:06:43.074539 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.074513    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-4d1799-b8845f4f5-mp8jg"]
Nov 03 22:06:43.904598 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.904564    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"]
Nov 03 22:06:43.905003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.904990    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing"
Nov 03 22:06:43.905049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905004    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing"
Nov 03 22:06:43.905049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905016    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing"
Nov 03 22:06:43.905049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905021    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing"
Nov 03 22:06:43.905049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905031    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6f61bb42-4c0a-4d95-92f7-687b8d2d6320" containerName="revoke-keys"
Nov 03 22:06:43.905049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905037    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f61bb42-4c0a-4d95-92f7-687b8d2d6320" containerName="revoke-keys"
Nov 03 22:06:43.905214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905129    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" containerName="payload-pre-processing"
Nov 03 22:06:43.905214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905136    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" containerName="payload-processing"
Nov 03 22:06:43.905214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.905143    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f61bb42-4c0a-4d95-92f7-687b8d2d6320" containerName="revoke-keys"
Nov 03 22:06:43.909552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.909530    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:43.962913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:43.962889    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"]
Nov 03 22:06:44.005971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.005946    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:06:44.006113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.005975    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:06:44.080392 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.080366    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.080542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.080424    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8f6\" (UniqueName: \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.080616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.080541    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.181769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.181737    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.181907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.181805    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8f6\" (UniqueName: \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.181907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.181860    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.182627 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.182603    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.184345 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.184316    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.240039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.240019    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8f6\" (UniqueName: \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\") pod \"maas-api-revoke-sub-9bcd1f14-hbczl\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.494646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.494561    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b006e79-d578-41a5-92c1-de0e5cf3551e" path="/var/lib/kubelet/pods/5b006e79-d578-41a5-92c1-de0e5cf3551e/volumes"
Nov 03 22:06:44.495139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.495119    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3558e41-3e6a-47c8-84a7-1b068ca3285d" path="/var/lib/kubelet/pods/f3558e41-3e6a-47c8-84a7-1b068ca3285d/volumes"
Nov 03 22:06:44.522697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.522674    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:44.901000 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:44.900967    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"]
Nov 03 22:06:44.901952 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:06:44.901926    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b0cd80_993f_44a5_b89d_fff411cf82b7.slice/crio-555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8 WatchSource:0}: Error finding container 555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8: Status 404 returned error can't find the container with id 555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8
Nov 03 22:06:45.431774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.431741    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" podUID="af25ef86-b58a-4835-99f7-1fb7215c69aa" containerName="main" containerID="cri-o://a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de" gracePeriod=45
Nov 03 22:06:45.704492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.704468    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:06:45.796269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796239    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796281    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796309    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\" (UniqueName: \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796382    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796476    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796521    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\") pod \"af25ef86-b58a-4835-99f7-1fb7215c69aa\" (UID: \"af25ef86-b58a-4835-99f7-1fb7215c69aa\") "
Nov 03 22:06:45.796800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796735    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:06:45.796912 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796815    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:06:45.796962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796903    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:06:45.797012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796965    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:45.797012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.796986    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:45.798581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.798556    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:06:45.798692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.798599    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:06:45.798692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.798659    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75" pod "af25ef86-b58a-4835-99f7-1fb7215c69aa" (UID: "af25ef86-b58a-4835-99f7-1fb7215c69aa"). InnerVolumeSpecName "kube-api-access-ngr75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:45.832270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.832245    5533 generic.go:356] "Generic (PLEG): container finished" podID="d2b0cd80-993f-44a5-b89d-fff411cf82b7" containerID="f3436f49aa49288f1ce9f9d9255595c50473ec00cb689e925b87876d2ff50b0d" exitCode=0
Nov 03 22:06:45.832380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.832324    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl" event={"ID":"d2b0cd80-993f-44a5-b89d-fff411cf82b7","Type":"ContainerDied","Data":"f3436f49aa49288f1ce9f9d9255595c50473ec00cb689e925b87876d2ff50b0d"}
Nov 03 22:06:45.832433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.832378    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl" event={"ID":"d2b0cd80-993f-44a5-b89d-fff411cf82b7","Type":"ContainerStarted","Data":"555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8"}
Nov 03 22:06:45.833707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.833689    5533 generic.go:356] "Generic (PLEG): container finished" podID="af25ef86-b58a-4835-99f7-1fb7215c69aa" containerID="a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de" exitCode=0
Nov 03 22:06:45.833789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.833714    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" event={"ID":"af25ef86-b58a-4835-99f7-1fb7215c69aa","Type":"ContainerDied","Data":"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"}
Nov 03 22:06:45.833789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.833735    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"
Nov 03 22:06:45.833789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.833759    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf" event={"ID":"af25ef86-b58a-4835-99f7-1fb7215c69aa","Type":"ContainerDied","Data":"18b12b1b499df4c2acc81ad26c21a372dd16c1fcad46a85d4b006508877a312e"}
Nov 03 22:06:45.833789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.833783    5533 scope.go:122] "RemoveContainer" containerID="a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"
Nov 03 22:06:45.849271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.849254    5533 scope.go:122] "RemoveContainer" containerID="a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"
Nov 03 22:06:45.849560 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:06:45.849541    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de\": container with ID starting with a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de not found: ID does not exist" containerID="a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"
Nov 03 22:06:45.849621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.849573    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de"} err="failed to get container status \"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de\": rpc error: code = NotFound desc = could not find container \"a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de\": container with ID starting with a5de64b632b41723af0816013b0736cd36540e69ca8f83804f59364bcb8af0de not found: ID does not exist"
Nov 03 22:06:45.898328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.898303    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:45.898328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.898327    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af25ef86-b58a-4835-99f7-1fb7215c69aa-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:45.898512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.898336    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af25ef86-b58a-4835-99f7-1fb7215c69aa-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:45.898512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:45.898346    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngr75\" (UniqueName: \"kubernetes.io/projected/af25ef86-b58a-4835-99f7-1fb7215c69aa-kube-api-access-ngr75\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:46.049049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:46.048978    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:06:46.058055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:46.058029    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-714979/rate-test-model-714979-kserve-7769cd9d65-lg7hf"]
Nov 03 22:06:46.492693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:46.492664    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af25ef86-b58a-4835-99f7-1fb7215c69aa" path="/var/lib/kubelet/pods/af25ef86-b58a-4835-99f7-1fb7215c69aa/volumes"
Nov 03 22:06:46.986720 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:46.986695    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:47.009191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.009167    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\") pod \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") "
Nov 03 22:06:47.009359 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.009344    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\") pod \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") "
Nov 03 22:06:47.009402 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.009375    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\" (UniqueName: \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\") pod \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\" (UID: \"d2b0cd80-993f-44a5-b89d-fff411cf82b7\") "
Nov 03 22:06:47.009715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.009688    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca" pod "d2b0cd80-993f-44a5-b89d-fff411cf82b7" (UID: "d2b0cd80-993f-44a5-b89d-fff411cf82b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:47.011322 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.011296    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca" pod "d2b0cd80-993f-44a5-b89d-fff411cf82b7" (UID: "d2b0cd80-993f-44a5-b89d-fff411cf82b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:06:47.011458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.011435    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6" pod "d2b0cd80-993f-44a5-b89d-fff411cf82b7" (UID: "d2b0cd80-993f-44a5-b89d-fff411cf82b7"). InnerVolumeSpecName "kube-api-access-dw8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:47.110992 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.110942    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw8f6\" (UniqueName: \"kubernetes.io/projected/d2b0cd80-993f-44a5-b89d-fff411cf82b7-kube-api-access-dw8f6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:47.110992 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.110965    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:47.110992 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.110976    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b0cd80-993f-44a5-b89d-fff411cf82b7-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:47.857974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.857946    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"
Nov 03 22:06:47.858347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.857984    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl" event={"ID":"d2b0cd80-993f-44a5-b89d-fff411cf82b7","Type":"ContainerDied","Data":"555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8"}
Nov 03 22:06:47.858347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:47.858033    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555cdb8605cb57e602a6a8f4d66fbbd526baeac99ed948f39dc347d98468bca8"
Nov 03 22:06:49.260897 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:06:49.260860    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:06:51.902280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:51.902245    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 22:06:52.013717 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:52.013680    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 22:06:54.712139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712097    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"]
Nov 03 22:06:54.712774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712751    5533 cpu_manager.go:414] "containerMap: removing container" podUID="af25ef86-b58a-4835-99f7-1fb7215c69aa" containerName="main"
Nov 03 22:06:54.712887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712775    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af25ef86-b58a-4835-99f7-1fb7215c69aa" containerName="main"
Nov 03 22:06:54.712887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712800    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d2b0cd80-993f-44a5-b89d-fff411cf82b7" containerName="revoke-keys"
Nov 03 22:06:54.712887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712808    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b0cd80-993f-44a5-b89d-fff411cf82b7" containerName="revoke-keys"
Nov 03 22:06:54.713051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712951    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="af25ef86-b58a-4835-99f7-1fb7215c69aa" containerName="main"
Nov 03 22:06:54.713051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.712966    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b0cd80-993f-44a5-b89d-fff411cf82b7" containerName="revoke-keys"
Nov 03 22:06:54.716534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.716513    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.773413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.773391    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"]
Nov 03 22:06:54.778299 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.778273    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.778432 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.778322    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.778432 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.778415    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqd7q\" (UniqueName: \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.807740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.807706    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:06:54.815843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.815822    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:06:54.879699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.879676    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.879823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.879722    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.879823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.879803    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqd7q\" (UniqueName: \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.880482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.880459    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.882168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.882143    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:54.937905 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:54.937881    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqd7q\" (UniqueName: \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\") pod \"maas-api-revoke-sub-876f9649-cnfvf\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:55.029309 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:55.029231    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:55.398920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:55.398889    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"]
Nov 03 22:06:55.400114 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:06:55.400085    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89711c04_b751_4817_8267_391f384c843d.slice/crio-2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b WatchSource:0}: Error finding container 2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b: Status 404 returned error can't find the container with id 2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b
Nov 03 22:06:55.908039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:55.908001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf" event={"ID":"89711c04-b751-4817-8267-391f384c843d","Type":"ContainerStarted","Data":"3ec5a52115e47b5847b19c37e1270be0b38b1517f5fc8f26a6b0a256d7fb0f7f"}
Nov 03 22:06:55.908381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:55.908065    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf" event={"ID":"89711c04-b751-4817-8267-391f384c843d","Type":"ContainerStarted","Data":"2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b"}
Nov 03 22:06:56.019731 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:56.019678    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf" podStartSLOduration=2.019665581 podStartE2EDuration="2.019665581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:56.016893243 +0000 UTC m=+4717.648206722" watchObservedRunningTime="2026-11-03 22:06:56.019665581 +0000 UTC m=+4717.650979085"
Nov 03 22:06:56.902844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:56.902799    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing" containerID="cri-o://247e9b2871799f4d520838845828eebef1110f62c104e84e036fab5ee5eeba2d" gracePeriod=25
Nov 03 22:06:56.915291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:56.915256    5533 generic.go:356] "Generic (PLEG): container finished" podID="89711c04-b751-4817-8267-391f384c843d" containerID="3ec5a52115e47b5847b19c37e1270be0b38b1517f5fc8f26a6b0a256d7fb0f7f" exitCode=0
Nov 03 22:06:56.915679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:56.915381    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf" event={"ID":"89711c04-b751-4817-8267-391f384c843d","Type":"ContainerDied","Data":"3ec5a52115e47b5847b19c37e1270be0b38b1517f5fc8f26a6b0a256d7fb0f7f"}
Nov 03 22:06:57.014671 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:57.014620    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing" containerID="cri-o://8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99" gracePeriod=25
Nov 03 22:06:58.078381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.078335    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:58.108086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.108057    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\" (UniqueName: \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\") pod \"89711c04-b751-4817-8267-391f384c843d\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") "
Nov 03 22:06:58.108237 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.108121    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\") pod \"89711c04-b751-4817-8267-391f384c843d\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") "
Nov 03 22:06:58.108237 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.108191    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\") pod \"89711c04-b751-4817-8267-391f384c843d\" (UID: \"89711c04-b751-4817-8267-391f384c843d\") "
Nov 03 22:06:58.109157 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.108558    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca" pod "89711c04-b751-4817-8267-391f384c843d" (UID: "89711c04-b751-4817-8267-391f384c843d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:06:58.110349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.110321    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q" pod "89711c04-b751-4817-8267-391f384c843d" (UID: "89711c04-b751-4817-8267-391f384c843d"). InnerVolumeSpecName "kube-api-access-nqd7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:06:58.110902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.110881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca" pod "89711c04-b751-4817-8267-391f384c843d" (UID: "89711c04-b751-4817-8267-391f384c843d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:06:58.209197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.209125    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89711c04-b751-4817-8267-391f384c843d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:58.209197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.209153    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqd7q\" (UniqueName: \"kubernetes.io/projected/89711c04-b751-4817-8267-391f384c843d-kube-api-access-nqd7q\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:58.209197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.209170    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89711c04-b751-4817-8267-391f384c843d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:06:58.349755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.349723    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:06:58.929303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.929272    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"
Nov 03 22:06:58.929494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.929305    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf" event={"ID":"89711c04-b751-4817-8267-391f384c843d","Type":"ContainerDied","Data":"2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b"}
Nov 03 22:06:58.929494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:06:58.929354    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2861601a0b48e96209d765ebb704f737e9cf413f13af5415de00d4d7aff7c85b"
Nov 03 22:07:00.672131 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:00.672096    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:01.682948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:01.682912    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:06.273385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.273348    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"]
Nov 03 22:07:06.273869 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.273854    5533 cpu_manager.go:414] "containerMap: removing container" podUID="89711c04-b751-4817-8267-391f384c843d" containerName="revoke-keys"
Nov 03 22:07:06.273925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.273871    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89711c04-b751-4817-8267-391f384c843d" containerName="revoke-keys"
Nov 03 22:07:06.273980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.273946    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="89711c04-b751-4817-8267-391f384c843d" containerName="revoke-keys"
Nov 03 22:07:06.277823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.277801    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.327420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.327395    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:07:06.373842 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.373815    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:07:06.376846 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.376814    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"]
Nov 03 22:07:06.385271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.385245    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.385391 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.385300    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvxc\" (UniqueName: \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.385482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.385443    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.486285 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.486252    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.486436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.486309    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvxc\" (UniqueName: \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.486436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.486366    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.487213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.487182    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.488550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.488529    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.543494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.543437    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvxc\" (UniqueName: \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\") pod \"maas-api-revoke-sub-c0a155d2-x9zmw\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.591182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.591159    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:06.957696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.957666    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"]
Nov 03 22:07:06.958807 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:07:06.958781    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9fb82f_d236_42e2_8615_14c7f0e05e96.slice/crio-e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be WatchSource:0}: Error finding container e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be: Status 404 returned error can't find the container with id e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be
Nov 03 22:07:06.980159 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:06.980134    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw" event={"ID":"9a9fb82f-d236-42e2-8615-14c7f0e05e96","Type":"ContainerStarted","Data":"e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be"}
Nov 03 22:07:07.988063 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:07.988034    5533 generic.go:356] "Generic (PLEG): container finished" podID="9a9fb82f-d236-42e2-8615-14c7f0e05e96" containerID="503bf974e5002c45820b9e9b5af3c07bd9d54aa6ca3cc991c782f8eb6361f66a" exitCode=0
Nov 03 22:07:07.988435 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:07.988122    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw" event={"ID":"9a9fb82f-d236-42e2-8615-14c7f0e05e96","Type":"ContainerDied","Data":"503bf974e5002c45820b9e9b5af3c07bd9d54aa6ca3cc991c782f8eb6361f66a"}
Nov 03 22:07:08.086293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:08.086263    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"]
Nov 03 22:07:08.088891 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:08.088869    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91741fe2-lrbtv"]
Nov 03 22:07:08.502154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:08.502123    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84426868-88e2-449c-8b1b-0a9568d586ef" path="/var/lib/kubelet/pods/84426868-88e2-449c-8b1b-0a9568d586ef/volumes"
Nov 03 22:07:09.131128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.131101    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:09.211039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.211014    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\") pod \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") "
Nov 03 22:07:09.211180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.211121    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\" (UniqueName: \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\") pod \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") "
Nov 03 22:07:09.211180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.211144    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\") pod \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\" (UID: \"9a9fb82f-d236-42e2-8615-14c7f0e05e96\") "
Nov 03 22:07:09.211360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.211335    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca" pod "9a9fb82f-d236-42e2-8615-14c7f0e05e96" (UID: "9a9fb82f-d236-42e2-8615-14c7f0e05e96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:09.211482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.211464    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:09.213216 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.213191    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca" pod "9a9fb82f-d236-42e2-8615-14c7f0e05e96" (UID: "9a9fb82f-d236-42e2-8615-14c7f0e05e96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:07:09.213341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.213317    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc" pod "9a9fb82f-d236-42e2-8615-14c7f0e05e96" (UID: "9a9fb82f-d236-42e2-8615-14c7f0e05e96"). InnerVolumeSpecName "kube-api-access-gcvxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:09.311996 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.311935    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvxc\" (UniqueName: \"kubernetes.io/projected/9a9fb82f-d236-42e2-8615-14c7f0e05e96-kube-api-access-gcvxc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:09.311996 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:09.311960    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9fb82f-d236-42e2-8615-14c7f0e05e96-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:10.002124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:10.002090    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw" event={"ID":"9a9fb82f-d236-42e2-8615-14c7f0e05e96","Type":"ContainerDied","Data":"e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be"}
Nov 03 22:07:10.002275 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:10.002139    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ef8668ccb0452ed1c715da850a320471ecc2921df2a3282a58e13d08a3b4be"
Nov 03 22:07:10.002275 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:10.002102    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"
Nov 03 22:07:10.672013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:10.671974    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:11.683105 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:11.683062    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:11.827571 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:11.827537    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 22:07:16.828089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:16.828031    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing" containerID="cri-o://31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66" gracePeriod=25
Nov 03 22:07:17.051010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.050979    5533 generic.go:356] "Generic (PLEG): container finished" podID="dbbc05dc-c298-4805-9710-1c770130d79d" containerID="247e9b2871799f4d520838845828eebef1110f62c104e84e036fab5ee5eeba2d" exitCode=0
Nov 03 22:07:17.051152 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.051012    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" event={"ID":"dbbc05dc-c298-4805-9710-1c770130d79d","Type":"ContainerDied","Data":"247e9b2871799f4d520838845828eebef1110f62c104e84e036fab5ee5eeba2d"}
Nov 03 22:07:17.168117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.168087    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 22:07:17.280068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.280043    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 22:07:17.287262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.287236    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\") pod \"dbbc05dc-c298-4805-9710-1c770130d79d\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") "
Nov 03 22:07:17.287390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.287350    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\" (UniqueName: \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\") pod \"dbbc05dc-c298-4805-9710-1c770130d79d\" (UID: \"dbbc05dc-c298-4805-9710-1c770130d79d\") "
Nov 03 22:07:17.287692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.287665    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume" pod "dbbc05dc-c298-4805-9710-1c770130d79d" (UID: "dbbc05dc-c298-4805-9710-1c770130d79d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:17.289310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.289288    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv" pod "dbbc05dc-c298-4805-9710-1c770130d79d" (UID: "dbbc05dc-c298-4805-9710-1c770130d79d"). InnerVolumeSpecName "kube-api-access-gt8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:17.376488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376461    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"]
Nov 03 22:07:17.376896 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376883    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing"
Nov 03 22:07:17.376949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376899    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing"
Nov 03 22:07:17.376949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376925    5533 cpu_manager.go:414] "containerMap: removing container" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing"
Nov 03 22:07:17.376949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376931    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing"
Nov 03 22:07:17.376949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376943    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9a9fb82f-d236-42e2-8615-14c7f0e05e96" containerName="revoke-keys"
Nov 03 22:07:17.376949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.376948    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9fb82f-d236-42e2-8615-14c7f0e05e96" containerName="revoke-keys"
Nov 03 22:07:17.377116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.377023    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" containerName="payload-processing"
Nov 03 22:07:17.377116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.377033    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9fb82f-d236-42e2-8615-14c7f0e05e96" containerName="revoke-keys"
Nov 03 22:07:17.377116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.377043    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerName="payload-pre-processing"
Nov 03 22:07:17.380479 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.380463    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.388800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.388780    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\") pod \"ac17cc0a-903d-47af-a5da-24f205a228e5\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") "
Nov 03 22:07:17.388884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.388827    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\" (UniqueName: \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\") pod \"ac17cc0a-903d-47af-a5da-24f205a228e5\" (UID: \"ac17cc0a-903d-47af-a5da-24f205a228e5\") "
Nov 03 22:07:17.389272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.389203    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume" pod "ac17cc0a-903d-47af-a5da-24f205a228e5" (UID: "ac17cc0a-903d-47af-a5da-24f205a228e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:17.389376 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.389363    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8kv\" (UniqueName: \"kubernetes.io/projected/dbbc05dc-c298-4805-9710-1c770130d79d-kube-api-access-gt8kv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:17.389433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.389379    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac17cc0a-903d-47af-a5da-24f205a228e5-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:17.389433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.389389    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbbc05dc-c298-4805-9710-1c770130d79d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:17.391149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.391127    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf" pod "ac17cc0a-903d-47af-a5da-24f205a228e5" (UID: "ac17cc0a-903d-47af-a5da-24f205a228e5"). InnerVolumeSpecName "kube-api-access-lr2rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:17.434383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.434352    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:07:17.434552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.434447    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:07:17.442257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.442178    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"]
Nov 03 22:07:17.490751 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.490712    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brq5p\" (UniqueName: \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.490918 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.490770    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.490963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.490921    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.491175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.491151    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2rf\" (UniqueName: \"kubernetes.io/projected/ac17cc0a-903d-47af-a5da-24f205a228e5-kube-api-access-lr2rf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:17.592568 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.592530    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.592777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.592654    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brq5p\" (UniqueName: \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.592777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.592688    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.593488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.593461    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.594969 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.594944    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.650281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.650206    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brq5p\" (UniqueName: \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\") pod \"maas-api-revoke-sub-181c96ac-tcxfj\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:17.693965 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:17.693934    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:18.060724 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.060679    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7" event={"ID":"dbbc05dc-c298-4805-9710-1c770130d79d","Type":"ContainerDied","Data":"e31f02719af6c1fad22f163c71e42b82193dab5d66c4eaac9ea8f2f8adcaa69f"}
Nov 03 22:07:18.061199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.060828    5533 scope.go:122] "RemoveContainer" containerID="247e9b2871799f4d520838845828eebef1110f62c104e84e036fab5ee5eeba2d"
Nov 03 22:07:18.061199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.061106    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"
Nov 03 22:07:18.072878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.071260    5533 generic.go:356] "Generic (PLEG): container finished" podID="ac17cc0a-903d-47af-a5da-24f205a228e5" containerID="8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99" exitCode=0
Nov 03 22:07:18.072878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.071423    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" event={"ID":"ac17cc0a-903d-47af-a5da-24f205a228e5","Type":"ContainerDied","Data":"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"}
Nov 03 22:07:18.072878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.071466    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"
Nov 03 22:07:18.072878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.071477    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7" event={"ID":"ac17cc0a-903d-47af-a5da-24f205a228e5","Type":"ContainerDied","Data":"2671f3d55b20dd61b1d8cc486070e1a412623aadae023e9075325a52d6b63bc4"}
Nov 03 22:07:18.081679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.081655    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"]
Nov 03 22:07:18.087407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.087384    5533 scope.go:122] "RemoveContainer" containerID="8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"
Nov 03 22:07:18.099696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.099673    5533 scope.go:122] "RemoveContainer" containerID="8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"
Nov 03 22:07:18.100005 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:07:18.099980    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99\": container with ID starting with 8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99 not found: ID does not exist" containerID="8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"
Nov 03 22:07:18.100097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.100014    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99"} err="failed to get container status \"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99\": rpc error: code = NotFound desc = could not find container \"8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99\": container with ID starting with 8ab0920a613eea1d72b5e567339efcf236824a2e2b0eaec5f08bd0a45b654b99 not found: ID does not exist"
Nov 03 22:07:18.217328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.217300    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 22:07:18.220188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.220162    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-714979-85d4c5469f-kbsr7"]
Nov 03 22:07:18.321289 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.321251    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 22:07:18.325281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.325253    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-714979-747cdcb78-d88m7"]
Nov 03 22:07:18.493985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.493916    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac17cc0a-903d-47af-a5da-24f205a228e5" path="/var/lib/kubelet/pods/ac17cc0a-903d-47af-a5da-24f205a228e5/volumes"
Nov 03 22:07:18.494331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:18.494317    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbc05dc-c298-4805-9710-1c770130d79d" path="/var/lib/kubelet/pods/dbbc05dc-c298-4805-9710-1c770130d79d/volumes"
Nov 03 22:07:19.052222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:19.052192    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"]
Nov 03 22:07:19.079887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:19.079859    5533 generic.go:356] "Generic (PLEG): container finished" podID="26d8ac77-64f9-47eb-95db-f929a67b7e0c" containerID="860e39f791119fd5d9ad81c7c24e8d4c6280274591a8b1837dbcabd9e94a5b1f" exitCode=0
Nov 03 22:07:19.080291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:19.079961    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj" event={"ID":"26d8ac77-64f9-47eb-95db-f929a67b7e0c","Type":"ContainerDied","Data":"860e39f791119fd5d9ad81c7c24e8d4c6280274591a8b1837dbcabd9e94a5b1f"}
Nov 03 22:07:19.080291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:19.080030    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj" event={"ID":"26d8ac77-64f9-47eb-95db-f929a67b7e0c","Type":"ContainerStarted","Data":"ddd5790612e7898b2164699bf221941cb81c0a9f3db56865dd627b937a89f702"}
Nov 03 22:07:19.098149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:19.098126    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c202087e-7mkj4"]
Nov 03 22:07:19.261744 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:07:19.261711    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:07:20.225448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.225422    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:20.319966 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.319942    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\") pod \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") "
Nov 03 22:07:20.320132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.319980    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\" (UniqueName: \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\") pod \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") "
Nov 03 22:07:20.320132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.320019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\") pod \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\" (UID: \"26d8ac77-64f9-47eb-95db-f929a67b7e0c\") "
Nov 03 22:07:20.320319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.320295    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca" pod "26d8ac77-64f9-47eb-95db-f929a67b7e0c" (UID: "26d8ac77-64f9-47eb-95db-f929a67b7e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:20.322029 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.322007    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca" pod "26d8ac77-64f9-47eb-95db-f929a67b7e0c" (UID: "26d8ac77-64f9-47eb-95db-f929a67b7e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:07:20.322111 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.322032    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p" pod "26d8ac77-64f9-47eb-95db-f929a67b7e0c" (UID: "26d8ac77-64f9-47eb-95db-f929a67b7e0c"). InnerVolumeSpecName "kube-api-access-brq5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:20.421734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.421708    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:20.421856 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.421736    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brq5p\" (UniqueName: \"kubernetes.io/projected/26d8ac77-64f9-47eb-95db-f929a67b7e0c-kube-api-access-brq5p\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:20.421856 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.421746    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d8ac77-64f9-47eb-95db-f929a67b7e0c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:20.494130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:20.494100    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08c5517-498b-4faa-ba09-dc1ffd2f69a3" path="/var/lib/kubelet/pods/f08c5517-498b-4faa-ba09-dc1ffd2f69a3/volumes"
Nov 03 22:07:21.097200 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:21.097170    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"
Nov 03 22:07:21.097352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:21.097214    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj" event={"ID":"26d8ac77-64f9-47eb-95db-f929a67b7e0c","Type":"ContainerDied","Data":"ddd5790612e7898b2164699bf221941cb81c0a9f3db56865dd627b937a89f702"}
Nov 03 22:07:21.097352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:21.097273    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd5790612e7898b2164699bf221941cb81c0a9f3db56865dd627b937a89f702"
Nov 03 22:07:21.989408 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:21.989375    5533 scope.go:122] "RemoveContainer" containerID="dd23d53e62a4828bfe74569a177c7a64202fae5f523d295c9bb462ecf9ac73b6"
Nov 03 22:07:22.036405 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:22.036364    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:25.322725 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.322691    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"]
Nov 03 22:07:25.323348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.323325    5533 cpu_manager.go:414] "containerMap: removing container" podUID="26d8ac77-64f9-47eb-95db-f929a67b7e0c" containerName="revoke-keys"
Nov 03 22:07:25.323448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.323348    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d8ac77-64f9-47eb-95db-f929a67b7e0c" containerName="revoke-keys"
Nov 03 22:07:25.323520 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.323508    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d8ac77-64f9-47eb-95db-f929a67b7e0c" containerName="revoke-keys"
Nov 03 22:07:25.328307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.328284    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.379704 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.379676    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:07:25.426390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.426368    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:07:25.429430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.429408    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"]
Nov 03 22:07:25.466245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.466222    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.466350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.466260    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.466468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.466449    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5g8\" (UniqueName: \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.567729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.567705    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5g8\" (UniqueName: \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.567839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.567751    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.567839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.567792    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.568421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.568395    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.570085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.570063    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.625441 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.625385    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5g8\" (UniqueName: \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\") pod \"maas-api-revoke-sub-59d972da-fxfb2\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:25.641030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:25.641005    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:26.007668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:26.007616    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"]
Nov 03 22:07:26.008914 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:07:26.008888    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4e5958_823c_4a13_a8eb_f39de41e3e43.slice/crio-2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379 WatchSource:0}: Error finding container 2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379: Status 404 returned error can't find the container with id 2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379
Nov 03 22:07:26.128135 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:26.128106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2" event={"ID":"1e4e5958-823c-4a13-a8eb-f39de41e3e43","Type":"ContainerStarted","Data":"2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379"}
Nov 03 22:07:27.136075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:27.135989    5533 generic.go:356] "Generic (PLEG): container finished" podID="1e4e5958-823c-4a13-a8eb-f39de41e3e43" containerID="051102faccd5c59c9318d4b290d3d456b64e9a8f9f69b321c0ee1e9cacca803e" exitCode=0
Nov 03 22:07:27.136075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:27.136023    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2" event={"ID":"1e4e5958-823c-4a13-a8eb-f39de41e3e43","Type":"ContainerDied","Data":"051102faccd5c59c9318d4b290d3d456b64e9a8f9f69b321c0ee1e9cacca803e"}
Nov 03 22:07:28.269076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.269050    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:28.397766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.397687    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\") pod \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") "
Nov 03 22:07:28.397907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.397785    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\") pod \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") "
Nov 03 22:07:28.397907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.397827    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\" (UniqueName: \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\") pod \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\" (UID: \"1e4e5958-823c-4a13-a8eb-f39de41e3e43\") "
Nov 03 22:07:28.398216 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.398172    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca" pod "1e4e5958-823c-4a13-a8eb-f39de41e3e43" (UID: "1e4e5958-823c-4a13-a8eb-f39de41e3e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:28.399772 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.399746    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca" pod "1e4e5958-823c-4a13-a8eb-f39de41e3e43" (UID: "1e4e5958-823c-4a13-a8eb-f39de41e3e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:07:28.399978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.399961    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8" pod "1e4e5958-823c-4a13-a8eb-f39de41e3e43" (UID: "1e4e5958-823c-4a13-a8eb-f39de41e3e43"). InnerVolumeSpecName "kube-api-access-df5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:28.499085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.499052    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:28.499085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.499083    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df5g8\" (UniqueName: \"kubernetes.io/projected/1e4e5958-823c-4a13-a8eb-f39de41e3e43-kube-api-access-df5g8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:28.499213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:28.499094    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4e5958-823c-4a13-a8eb-f39de41e3e43-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:29.149869 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:29.149836    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"
Nov 03 22:07:29.150044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:29.149869    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2" event={"ID":"1e4e5958-823c-4a13-a8eb-f39de41e3e43","Type":"ContainerDied","Data":"2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379"}
Nov 03 22:07:29.150044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:29.149919    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2845eeb2688de720171fe44fc7e2560feb6065f0db66674675c10419fc076379"
Nov 03 22:07:30.047338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:30.047302    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"]
Nov 03 22:07:30.093564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:30.093532    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86089293-r5cfx"]
Nov 03 22:07:30.492914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:30.492884    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ada136-7ec5-4bdf-b2c2-33686cf87adc" path="/var/lib/kubelet/pods/89ada136-7ec5-4bdf-b2c2-33686cf87adc/volumes"
Nov 03 22:07:32.037269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:32.037225    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:35.990695 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:35.990661    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:07:37.081052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.081027    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 22:07:37.180897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.180875    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\") pod \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") "
Nov 03 22:07:37.181022 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.180986    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\" (UniqueName: \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\") pod \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\" (UID: \"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9\") "
Nov 03 22:07:37.181262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.181237    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume" pod "1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" (UID: "1041fb5d-f0c2-4c6d-a78a-8829044ecdb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:37.183007 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.182978    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm" pod "1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" (UID: "1041fb5d-f0c2-4c6d-a78a-8829044ecdb9"). InnerVolumeSpecName "kube-api-access-q57sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:37.209055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.209030    5533 generic.go:356] "Generic (PLEG): container finished" podID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerID="31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66" exitCode=0
Nov 03 22:07:37.209173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.209073    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"
Nov 03 22:07:37.209173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.209106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" event={"ID":"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9","Type":"ContainerDied","Data":"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"}
Nov 03 22:07:37.209173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.209161    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb" event={"ID":"1041fb5d-f0c2-4c6d-a78a-8829044ecdb9","Type":"ContainerDied","Data":"def5c01fbb822b21c0ae4559cae5cb4318c2f2ca725acdc262336a8d31fa9602"}
Nov 03 22:07:37.209290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.209183    5533 scope.go:122] "RemoveContainer" containerID="31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"
Nov 03 22:07:37.225477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.225454    5533 scope.go:122] "RemoveContainer" containerID="31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"
Nov 03 22:07:37.225869 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:07:37.225838    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66\": container with ID starting with 31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66 not found: ID does not exist" containerID="31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"
Nov 03 22:07:37.225969 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.225880    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66"} err="failed to get container status \"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66\": rpc error: code = NotFound desc = could not find container \"31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66\": container with ID starting with 31d2d328fe97a9ccd305aae053fa2687142954b680db537ebf3e01a6bb231b66 not found: ID does not exist"
Nov 03 22:07:37.282371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.282346    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:37.282371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.282370    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57sm\" (UniqueName: \"kubernetes.io/projected/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9-kube-api-access-q57sm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:37.364464 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.364440    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 22:07:37.367870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.367845    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9818d8-5797b7d4d-2hvfb"]
Nov 03 22:07:37.445731 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.445671    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"]
Nov 03 22:07:37.446153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446140    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1e4e5958-823c-4a13-a8eb-f39de41e3e43" containerName="revoke-keys"
Nov 03 22:07:37.446197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446155    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4e5958-823c-4a13-a8eb-f39de41e3e43" containerName="revoke-keys"
Nov 03 22:07:37.446197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446172    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing"
Nov 03 22:07:37.446197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446177    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing"
Nov 03 22:07:37.446297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446249    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" containerName="payload-processing"
Nov 03 22:07:37.446297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.446263    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4e5958-823c-4a13-a8eb-f39de41e3e43" containerName="revoke-keys"
Nov 03 22:07:37.450700 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.450684    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.504916 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.504889    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"]
Nov 03 22:07:37.506841 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.506821    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:07:37.507008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.506829    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:07:37.585332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.585304    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.585452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.585374    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.585452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.585440    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbtq\" (UniqueName: \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.686547 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.686519    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbtq\" (UniqueName: \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.686676 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.686610    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.686743 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.686718    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.687348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.687323    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.689080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.689053    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.744348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.744296    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbtq\" (UniqueName: \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\") pod \"maas-api-revoke-sub-edb9ef22-rrb6x\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:37.763800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:37.763781    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:38.131872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:38.131841    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"]
Nov 03 22:07:38.132922 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:07:38.132895    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea81498_b4ba_468e_927d_60f41941414a.slice/crio-91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0 WatchSource:0}: Error finding container 91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0: Status 404 returned error can't find the container with id 91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0
Nov 03 22:07:38.217501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:38.217474    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x" event={"ID":"0ea81498-b4ba-468e-927d-60f41941414a","Type":"ContainerStarted","Data":"91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0"}
Nov 03 22:07:38.491973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:38.491949    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1041fb5d-f0c2-4c6d-a78a-8829044ecdb9" path="/var/lib/kubelet/pods/1041fb5d-f0c2-4c6d-a78a-8829044ecdb9/volumes"
Nov 03 22:07:39.225593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:39.225561    5533 generic.go:356] "Generic (PLEG): container finished" podID="0ea81498-b4ba-468e-927d-60f41941414a" containerID="747a8e514a724a3b21c9abd21a18b4ced1d10ad6a12c88d7cda44c2c3a605a3a" exitCode=0
Nov 03 22:07:39.225993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:39.225606    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x" event={"ID":"0ea81498-b4ba-468e-927d-60f41941414a","Type":"ContainerDied","Data":"747a8e514a724a3b21c9abd21a18b4ced1d10ad6a12c88d7cda44c2c3a605a3a"}
Nov 03 22:07:40.378201 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.378172    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:40.411356 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.411330    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\") pod \"0ea81498-b4ba-468e-927d-60f41941414a\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") "
Nov 03 22:07:40.411483 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.411387    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\" (UniqueName: \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\") pod \"0ea81498-b4ba-468e-927d-60f41941414a\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") "
Nov 03 22:07:40.411483 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.411437    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\") pod \"0ea81498-b4ba-468e-927d-60f41941414a\" (UID: \"0ea81498-b4ba-468e-927d-60f41941414a\") "
Nov 03 22:07:40.411850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.411829    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca" pod "0ea81498-b4ba-468e-927d-60f41941414a" (UID: "0ea81498-b4ba-468e-927d-60f41941414a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:07:40.413226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.413204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca" pod "0ea81498-b4ba-468e-927d-60f41941414a" (UID: "0ea81498-b4ba-468e-927d-60f41941414a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:07:40.413405 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.413388    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq" pod "0ea81498-b4ba-468e-927d-60f41941414a" (UID: "0ea81498-b4ba-468e-927d-60f41941414a"). InnerVolumeSpecName "kube-api-access-cdbtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:07:40.512406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.512379    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:40.512525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.512409    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbtq\" (UniqueName: \"kubernetes.io/projected/0ea81498-b4ba-468e-927d-60f41941414a-kube-api-access-cdbtq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:40.512525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:40.512423    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea81498-b4ba-468e-927d-60f41941414a-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:07:41.239070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:41.239038    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"
Nov 03 22:07:41.239070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:41.239055    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x" event={"ID":"0ea81498-b4ba-468e-927d-60f41941414a","Type":"ContainerDied","Data":"91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0"}
Nov 03 22:07:41.239298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:41.239113    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d1bc205f1ba9da4422b7418d9e8c499a91f6a7cb996b88085c6c2088d3dfd0"
Nov 03 22:07:42.000185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:42.000146    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 22:07:42.045551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:42.045495    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"]
Nov 03 22:07:42.048536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:42.048507    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0344651c-7hgp8"]
Nov 03 22:07:42.113915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:42.113885    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 22:07:42.493238 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:42.493209    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc01ffb6-be22-4e59-884d-c013f937b610" path="/var/lib/kubelet/pods/fc01ffb6-be22-4e59-884d-c013f937b610/volumes"
Nov 03 22:07:47.000570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:47.000528    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing" containerID="cri-o://39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce" gracePeriod=25
Nov 03 22:07:47.114760 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:47.114711    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing" containerID="cri-o://78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57" gracePeriod=25
Nov 03 22:07:49.261153 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:07:49.261115    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:07:52.189930 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:52.189885    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:52.190428 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:52.190072    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:07:57.276956 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:07:57.276924    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:08:02.189380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:02.189295    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:08:02.189380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:02.189353    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:08:07.252257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.252230    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 22:08:07.361578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.361545    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\") pod \"f165d751-3520-4a6f-913a-107b290ed114\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") "
Nov 03 22:08:07.361744 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.361670    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\" (UniqueName: \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\") pod \"f165d751-3520-4a6f-913a-107b290ed114\" (UID: \"f165d751-3520-4a6f-913a-107b290ed114\") "
Nov 03 22:08:07.362360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.362299    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume" pod "f165d751-3520-4a6f-913a-107b290ed114" (UID: "f165d751-3520-4a6f-913a-107b290ed114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:07.365559 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.365521    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q" pod "f165d751-3520-4a6f-913a-107b290ed114" (UID: "f165d751-3520-4a6f-913a-107b290ed114"). InnerVolumeSpecName "kube-api-access-gbf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:07.367182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.367160    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 22:08:07.416884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.416852    5533 generic.go:356] "Generic (PLEG): container finished" podID="f165d751-3520-4a6f-913a-107b290ed114" containerID="39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce" exitCode=0
Nov 03 22:08:07.417044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.416910    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"
Nov 03 22:08:07.417044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.416939    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" event={"ID":"f165d751-3520-4a6f-913a-107b290ed114","Type":"ContainerDied","Data":"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"}
Nov 03 22:08:07.417044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.416993    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v" event={"ID":"f165d751-3520-4a6f-913a-107b290ed114","Type":"ContainerDied","Data":"4136a26ddf71e51e9baf6967ee6084970af202f64588b670af3fc601d22d078e"}
Nov 03 22:08:07.417044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.417021    5533 scope.go:122] "RemoveContainer" containerID="39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"
Nov 03 22:08:07.418496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.418463    5533 generic.go:356] "Generic (PLEG): container finished" podID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerID="78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57" exitCode=0
Nov 03 22:08:07.418620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.418522    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"
Nov 03 22:08:07.418620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.418517    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" event={"ID":"cda9d121-a1e5-4398-bd9d-d0b12700dac6","Type":"ContainerDied","Data":"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"}
Nov 03 22:08:07.418788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.418677    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5" event={"ID":"cda9d121-a1e5-4398-bd9d-d0b12700dac6","Type":"ContainerDied","Data":"3f7050df661d2da52b2bf038921cbba85252f2e337232966296a4dc0effabc19"}
Nov 03 22:08:07.427449 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.427428    5533 scope.go:122] "RemoveContainer" containerID="39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"
Nov 03 22:08:07.427719 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:07.427699    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce\": container with ID starting with 39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce not found: ID does not exist" containerID="39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"
Nov 03 22:08:07.427777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.427729    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce"} err="failed to get container status \"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce\": rpc error: code = NotFound desc = could not find container \"39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce\": container with ID starting with 39b1530250768befa3002e03134918cfc4591fdb738f0fecb52513a3fce60dce not found: ID does not exist"
Nov 03 22:08:07.427777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.427747    5533 scope.go:122] "RemoveContainer" containerID="78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"
Nov 03 22:08:07.436571 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.436551    5533 scope.go:122] "RemoveContainer" containerID="78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"
Nov 03 22:08:07.436832 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:07.436814    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57\": container with ID starting with 78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57 not found: ID does not exist" containerID="78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"
Nov 03 22:08:07.436892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.436840    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57"} err="failed to get container status \"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57\": rpc error: code = NotFound desc = could not find container \"78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57\": container with ID starting with 78ca97c35ff69c11cf932e7be3baf29a82471dee6825318ab6286098b63b2b57 not found: ID does not exist"
Nov 03 22:08:07.463202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463180    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\") pod \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") "
Nov 03 22:08:07.463333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463316    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\" (UniqueName: \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\") pod \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\" (UID: \"cda9d121-a1e5-4398-bd9d-d0b12700dac6\") "
Nov 03 22:08:07.463572 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463552    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume" pod "cda9d121-a1e5-4398-bd9d-d0b12700dac6" (UID: "cda9d121-a1e5-4398-bd9d-d0b12700dac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:07.463655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463624    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f165d751-3520-4a6f-913a-107b290ed114-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:07.463702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463652    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda9d121-a1e5-4398-bd9d-d0b12700dac6-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:07.463702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.463663    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbf9q\" (UniqueName: \"kubernetes.io/projected/f165d751-3520-4a6f-913a-107b290ed114-kube-api-access-gbf9q\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:07.465197 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.465177    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss" pod "cda9d121-a1e5-4398-bd9d-d0b12700dac6" (UID: "cda9d121-a1e5-4398-bd9d-d0b12700dac6"). InnerVolumeSpecName "kube-api-access-8vbss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:07.564915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.564833    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbss\" (UniqueName: \"kubernetes.io/projected/cda9d121-a1e5-4398-bd9d-d0b12700dac6-kube-api-access-8vbss\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:07.632690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.632656    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 22:08:07.639693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.639666    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a6219b-54b6984d8d-gr97v"]
Nov 03 22:08:07.834083 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.834003    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 22:08:07.838484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:07.838458    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a6219b-8487557659-clnv5"]
Nov 03 22:08:08.493763 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:08.493729    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" path="/var/lib/kubelet/pods/cda9d121-a1e5-4398-bd9d-d0b12700dac6/volumes"
Nov 03 22:08:08.494261 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:08.494095    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f165d751-3520-4a6f-913a-107b290ed114" path="/var/lib/kubelet/pods/f165d751-3520-4a6f-913a-107b290ed114/volumes"
Nov 03 22:08:13.661029 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.660994    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661430    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing"
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661443    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing"
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661452    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0ea81498-b4ba-468e-927d-60f41941414a" containerName="revoke-keys"
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661458    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea81498-b4ba-468e-927d-60f41941414a" containerName="revoke-keys"
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661475    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing"
Nov 03 22:08:13.661527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661480    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing"
Nov 03 22:08:13.661901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661560    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f165d751-3520-4a6f-913a-107b290ed114" containerName="payload-processing"
Nov 03 22:08:13.661901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661570    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea81498-b4ba-468e-927d-60f41941414a" containerName="revoke-keys"
Nov 03 22:08:13.661901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.661578    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda9d121-a1e5-4398-bd9d-d0b12700dac6" containerName="payload-pre-processing"
Nov 03 22:08:13.666389 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.666366    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.721590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.721557    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a41ad0-openshift-default-dockercfg-dhkbv\""
Nov 03 22:08:13.771242 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.771204    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:13.818962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.818933    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.818971    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.818995    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819153    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819190    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5qr\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819215    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819272    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819408 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819324    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.819463 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.819426    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.920972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.920944    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921138 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.920984    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5qr\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921138 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921003    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921138 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921115    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921351 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921183    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921351 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921246    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921351 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921292    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.921976 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921316    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921979    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921805    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921928    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.921727    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.922024    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.922338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.922249    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.923473 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.923453    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.923612 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.923595    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.978696 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.978669    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5qr\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.979271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.979248    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\") pod \"e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") " pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:13.980089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:13.980067    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:14.405554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.405527    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:14.406902 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:08:14.406870    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c1e07e_0efd_4b6c_9d31_71e689a0769b.slice/crio-110e1fcada01a88e974ff06770a87d080abd5d7ecd8a7044b1e312170095660c WatchSource:0}: Error finding container 110e1fcada01a88e974ff06770a87d080abd5d7ecd8a7044b1e312170095660c: Status 404 returned error can't find the container with id 110e1fcada01a88e974ff06770a87d080abd5d7ecd8a7044b1e312170095660c
Nov 03 22:08:14.409209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.409178    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:08:14.409310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.409240    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:08:14.409310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.409270    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:08:14.468979 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.468946    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" event={"ID":"95c1e07e-0efd-4b6c-9d31-71e689a0769b","Type":"ContainerStarted","Data":"110e1fcada01a88e974ff06770a87d080abd5d7ecd8a7044b1e312170095660c"}
Nov 03 22:08:14.587578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.587521    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" podStartSLOduration=1.587507994 podStartE2EDuration="1.587507994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:14.585720425 +0000 UTC m=+4796.217033903" watchObservedRunningTime="2026-11-03 22:08:14.587507994 +0000 UTC m=+4796.218821470"
Nov 03 22:08:14.673299 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.673265    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"]
Nov 03 22:08:14.677706 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.677684    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.727656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.727612    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:08:14.727656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.727612    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:08:14.732176 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.732153    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"]
Nov 03 22:08:14.828810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.828776    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.828970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.828824    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.828970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.828937    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrc7\" (UniqueName: \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.930031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.929951    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.930031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.929994    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.930271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.930041    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrc7\" (UniqueName: \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.930735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.930709    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.932535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.932515    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.980535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.980514    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:14.987758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.987730    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrc7\" (UniqueName: \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\") pod \"maas-api-revoke-sub-34c9d3d9-jgtdl\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:14.991509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:14.991493    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:15.389704 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:15.370613    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"]
Nov 03 22:08:15.389704 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:08:15.378745    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e781bf4_58fd_4653_8dc1_5daa300e12e5.slice/crio-76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595 WatchSource:0}: Error finding container 76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595: Status 404 returned error can't find the container with id 76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595
Nov 03 22:08:15.491062 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:15.491018    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl" event={"ID":"1e781bf4-58fd-4653-8dc1-5daa300e12e5","Type":"ContainerStarted","Data":"76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595"}
Nov 03 22:08:15.493300 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:15.493248    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" event={"ID":"95c1e07e-0efd-4b6c-9d31-71e689a0769b","Type":"ContainerStarted","Data":"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"}
Nov 03 22:08:15.982003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:15.981910    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 22:08:15.982003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:15.981972    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 22:08:16.501313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:16.501281    5533 generic.go:356] "Generic (PLEG): container finished" podID="1e781bf4-58fd-4653-8dc1-5daa300e12e5" containerID="f755fc5d6d5b1a52c9a3b0315efafcc4515bac3bfedc6d3b2020bd0a68e067d4" exitCode=0
Nov 03 22:08:16.501460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:16.501370    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl" event={"ID":"1e781bf4-58fd-4653-8dc1-5daa300e12e5","Type":"ContainerDied","Data":"f755fc5d6d5b1a52c9a3b0315efafcc4515bac3bfedc6d3b2020bd0a68e067d4"}
Nov 03 22:08:16.981142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:16.981108    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 22:08:16.981345 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:16.981162    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 22:08:17.706568 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.706540    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:17.883603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.883533    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\" (UniqueName: \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\") pod \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") "
Nov 03 22:08:17.883603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.883594    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\") pod \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") "
Nov 03 22:08:17.883832 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.883657    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\") pod \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\" (UID: \"1e781bf4-58fd-4653-8dc1-5daa300e12e5\") "
Nov 03 22:08:17.884052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.884027    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca" pod "1e781bf4-58fd-4653-8dc1-5daa300e12e5" (UID: "1e781bf4-58fd-4653-8dc1-5daa300e12e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:17.885740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.885711    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca" pod "1e781bf4-58fd-4653-8dc1-5daa300e12e5" (UID: "1e781bf4-58fd-4653-8dc1-5daa300e12e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:08:17.885861 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.885768    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7" pod "1e781bf4-58fd-4653-8dc1-5daa300e12e5" (UID: "1e781bf4-58fd-4653-8dc1-5daa300e12e5"). InnerVolumeSpecName "kube-api-access-ftrc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:17.981053 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.981017    5533 patch_prober.go:29] interesting pod/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body=
Nov 03 22:08:17.981214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.981069    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused"
Nov 03 22:08:17.984525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.984502    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.984525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.984525    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrc7\" (UniqueName: \"kubernetes.io/projected/1e781bf4-58fd-4653-8dc1-5daa300e12e5-kube-api-access-ftrc7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.984677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:17.984537    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e781bf4-58fd-4653-8dc1-5daa300e12e5-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.392099 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:18.392058    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:18.392430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:18.392293    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy" containerID="cri-o://8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646" gracePeriod=30
Nov 03 22:08:18.518252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:18.516820    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl" event={"ID":"1e781bf4-58fd-4653-8dc1-5daa300e12e5","Type":"ContainerDied","Data":"76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595"}
Nov 03 22:08:18.518252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:18.516873    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76f2a65f054c4b0635560a500050613329d202126804204cfa52cdf15f4ac595"
Nov 03 22:08:18.518252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:18.516935    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"
Nov 03 22:08:19.079777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:19.079741    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 22:08:19.293648 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:19.293592    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:08:19.402184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:19.402101    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:08:19.415993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:19.415964    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:08:19.432453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:19.432428    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:08:19.443779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:19.443758    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:08:22.045259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:22.045213    5533 scope.go:122] "RemoveContainer" containerID="aa2ac85645b4984230e6c9c133e7e70403198f95c7b87ab34e6c603c1f8bc11c"
Nov 03 22:08:22.059148 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:22.059117    5533 scope.go:122] "RemoveContainer" containerID="070909469fce4af1d97cb110b0c7dc30ef774c52c1b38dd6a40ee11180056da3"
Nov 03 22:08:27.121675 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:27.121615    5533 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c1e07e_0efd_4b6c_9d31_71e689a0769b.slice/crio-conmon-8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 22:08:27.270126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.270097    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:27.369908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.369834    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.369908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.369882    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.369946    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.369971    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370009    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370049    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370090    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370397 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370142    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370397 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370183    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\") pod \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\" (UID: \"95c1e07e-0efd-4b6c-9d31-71e689a0769b\") "
Nov 03 22:08:27.370536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370506    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:27.370888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370691    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:27.370888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370822    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:27.370888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.370876    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:27.371179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.371118    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:27.372536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.372510    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "kube-api-access-4t5qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:27.372707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.372686    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:27.373036 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.373016    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:27.373337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.373316    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token" pod "95c1e07e-0efd-4b6c-9d31-71e689a0769b" (UID: "95c1e07e-0efd-4b6c-9d31-71e689a0769b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:27.471913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471877    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.471913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471906    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.471913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471916    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471925    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471934    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5qr\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-kube-api-access-4t5qr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471944    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471953    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471961    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c1e07e-0efd-4b6c-9d31-71e689a0769b-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.472144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.471969    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c1e07e-0efd-4b6c-9d31-71e689a0769b-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:27.587350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.587316    5533 generic.go:356] "Generic (PLEG): container finished" podID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerID="8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646" exitCode=0
Nov 03 22:08:27.587502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.587360    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"
Nov 03 22:08:27.587502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.587405    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" event={"ID":"95c1e07e-0efd-4b6c-9d31-71e689a0769b","Type":"ContainerDied","Data":"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"}
Nov 03 22:08:27.587502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.587457    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58" event={"ID":"95c1e07e-0efd-4b6c-9d31-71e689a0769b","Type":"ContainerDied","Data":"110e1fcada01a88e974ff06770a87d080abd5d7ecd8a7044b1e312170095660c"}
Nov 03 22:08:27.587502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.587479    5533 scope.go:122] "RemoveContainer" containerID="8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"
Nov 03 22:08:27.597527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.597510    5533 scope.go:122] "RemoveContainer" containerID="8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"
Nov 03 22:08:27.597873 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:27.597852    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646\": container with ID starting with 8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646 not found: ID does not exist" containerID="8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"
Nov 03 22:08:27.597946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.597879    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646"} err="failed to get container status \"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646\": rpc error: code = NotFound desc = could not find container \"8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646\": container with ID starting with 8d19d0acc7e25fae77743959afc006c0624d29f77c893d77428005f0f431a646 not found: ID does not exist"
Nov 03 22:08:27.753646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.753613    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:27.775714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.775690    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a41ad0-openshift-default-6c4db55cb9-mfv58"]
Nov 03 22:08:27.778090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.778070    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 22:08:27.833501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:27.833477    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 22:08:28.494669 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:28.494612    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" path="/var/lib/kubelet/pods/95c1e07e-0efd-4b6c-9d31-71e689a0769b/volumes"
Nov 03 22:08:30.967364 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:30.967330    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:08:31.318686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.318599    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"]
Nov 03 22:08:31.319063 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319050    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1e781bf4-58fd-4653-8dc1-5daa300e12e5" containerName="revoke-keys"
Nov 03 22:08:31.319112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319065    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e781bf4-58fd-4653-8dc1-5daa300e12e5" containerName="revoke-keys"
Nov 03 22:08:31.319112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319093    5533 cpu_manager.go:414] "containerMap: removing container" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy"
Nov 03 22:08:31.319112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319098    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy"
Nov 03 22:08:31.319211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319186    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e781bf4-58fd-4653-8dc1-5daa300e12e5" containerName="revoke-keys"
Nov 03 22:08:31.319211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.319197    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c1e07e-0efd-4b6c-9d31-71e689a0769b" containerName="istio-proxy"
Nov 03 22:08:31.322479 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.322461    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.377622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.377584    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"]
Nov 03 22:08:31.410678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.410629    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgzf\" (UniqueName: \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.410804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.410721    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.410804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.410797    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.419833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.419810    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:08:31.419951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.419830    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:08:31.511877 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.511852    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgzf\" (UniqueName: \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.512008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.511885    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.512008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.511936    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.512552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.512531    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.514363 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.514338    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.569562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.569504    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgzf\" (UniqueName: \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\") pod \"maas-api-revoke-sub-d053e827-nsvqq\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:31.635698 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:31.635650    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:32.013450 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.013418    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"]
Nov 03 22:08:32.014706 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:08:32.014667    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cdf215_1c68_4bcf_a11d_9c19f78b9c1e.slice/crio-f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8 WatchSource:0}: Error finding container f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8: Status 404 returned error can't find the container with id f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8
Nov 03 22:08:32.571232 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.571144    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 22:08:32.623487 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.623455    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq" event={"ID":"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e","Type":"ContainerStarted","Data":"171668d797123744d43c8aa675de38b6e1667f2d97153b7c5cc465f8316d0f03"}
Nov 03 22:08:32.623662 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.623508    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq" event={"ID":"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e","Type":"ContainerStarted","Data":"f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8"}
Nov 03 22:08:32.681280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.681240    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 22:08:32.778585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.778551    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" podUID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" containerName="payload-processing" containerID="cri-o://cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84" gracePeriod=25
Nov 03 22:08:32.793529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.793488    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq" podStartSLOduration=1.79347558 podStartE2EDuration="1.79347558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:08:32.792525003 +0000 UTC m=+4814.423838481" watchObservedRunningTime="2026-11-03 22:08:32.79347558 +0000 UTC m=+4814.424789102"
Nov 03 22:08:32.834100 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:32.834045    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" podUID="f5c811d2-bf9e-4076-b729-23324a24d3c8" containerName="payload-pre-processing" containerID="cri-o://df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f" gracePeriod=25
Nov 03 22:08:33.107724 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.107690    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 22:08:33.123762 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.123734    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 22:08:33.228764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.228733    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\") pod \"f5c811d2-bf9e-4076-b729-23324a24d3c8\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") "
Nov 03 22:08:33.228952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.228808    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\") pod \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") "
Nov 03 22:08:33.228952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.228910    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\" (UniqueName: \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\") pod \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\" (UID: \"7471ab21-70ed-4fb1-91c4-84caadfeb05a\") "
Nov 03 22:08:33.228952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.228931    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\" (UniqueName: \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\") pod \"f5c811d2-bf9e-4076-b729-23324a24d3c8\" (UID: \"f5c811d2-bf9e-4076-b729-23324a24d3c8\") "
Nov 03 22:08:33.229215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.229181    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume" pod "f5c811d2-bf9e-4076-b729-23324a24d3c8" (UID: "f5c811d2-bf9e-4076-b729-23324a24d3c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:33.229215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.229180    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume" pod "7471ab21-70ed-4fb1-91c4-84caadfeb05a" (UID: "7471ab21-70ed-4fb1-91c4-84caadfeb05a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:33.231058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.231031    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6" pod "f5c811d2-bf9e-4076-b729-23324a24d3c8" (UID: "f5c811d2-bf9e-4076-b729-23324a24d3c8"). InnerVolumeSpecName "kube-api-access-69rq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:33.231194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.231030    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl" pod "7471ab21-70ed-4fb1-91c4-84caadfeb05a" (UID: "7471ab21-70ed-4fb1-91c4-84caadfeb05a"). InnerVolumeSpecName "kube-api-access-6bcjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:33.329516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.329486    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bcjl\" (UniqueName: \"kubernetes.io/projected/7471ab21-70ed-4fb1-91c4-84caadfeb05a-kube-api-access-6bcjl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:33.329516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.329516    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69rq6\" (UniqueName: \"kubernetes.io/projected/f5c811d2-bf9e-4076-b729-23324a24d3c8-kube-api-access-69rq6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:33.329711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.329532    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c811d2-bf9e-4076-b729-23324a24d3c8-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:33.329711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.329545    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7471ab21-70ed-4fb1-91c4-84caadfeb05a-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:33.631572 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.631537    5533 generic.go:356] "Generic (PLEG): container finished" podID="f5c811d2-bf9e-4076-b729-23324a24d3c8" containerID="df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f" exitCode=0
Nov 03 22:08:33.631729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.631585    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"
Nov 03 22:08:33.631729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.631619    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" event={"ID":"f5c811d2-bf9e-4076-b729-23324a24d3c8","Type":"ContainerDied","Data":"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"}
Nov 03 22:08:33.631729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.631688    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx" event={"ID":"f5c811d2-bf9e-4076-b729-23324a24d3c8","Type":"ContainerDied","Data":"8059303dc97ca7058bb6c637c2f154a79bf8cb6df87cccc372ceff987bf9ae5e"}
Nov 03 22:08:33.631729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.631711    5533 scope.go:122] "RemoveContainer" containerID="df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"
Nov 03 22:08:33.632967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.632946    5533 generic.go:356] "Generic (PLEG): container finished" podID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" containerID="cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84" exitCode=0
Nov 03 22:08:33.633113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.632995    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"
Nov 03 22:08:33.633113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.633000    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" event={"ID":"7471ab21-70ed-4fb1-91c4-84caadfeb05a","Type":"ContainerDied","Data":"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"}
Nov 03 22:08:33.633113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.633053    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn" event={"ID":"7471ab21-70ed-4fb1-91c4-84caadfeb05a","Type":"ContainerDied","Data":"79d3afe2960147b7ad688fb63a4108fa1b0fa3249fb276477997c22faa7cacc7"}
Nov 03 22:08:33.634800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.634778    5533 generic.go:356] "Generic (PLEG): container finished" podID="f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" containerID="171668d797123744d43c8aa675de38b6e1667f2d97153b7c5cc465f8316d0f03" exitCode=0
Nov 03 22:08:33.634914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.634838    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq" event={"ID":"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e","Type":"ContainerDied","Data":"171668d797123744d43c8aa675de38b6e1667f2d97153b7c5cc465f8316d0f03"}
Nov 03 22:08:33.643352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.643336    5533 scope.go:122] "RemoveContainer" containerID="df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"
Nov 03 22:08:33.643625 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:33.643606    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f\": container with ID starting with df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f not found: ID does not exist" containerID="df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"
Nov 03 22:08:33.643738 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.643643    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f"} err="failed to get container status \"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f\": rpc error: code = NotFound desc = could not find container \"df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f\": container with ID starting with df60498f93056b9da4e6526b8b64e996e8cf0b07c24af05b1759b6e093b6138f not found: ID does not exist"
Nov 03 22:08:33.643738 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.643662    5533 scope.go:122] "RemoveContainer" containerID="cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"
Nov 03 22:08:33.652551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.652529    5533 scope.go:122] "RemoveContainer" containerID="cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"
Nov 03 22:08:33.652851 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:33.652832    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84\": container with ID starting with cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84 not found: ID does not exist" containerID="cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"
Nov 03 22:08:33.652942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.652855    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84"} err="failed to get container status \"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84\": rpc error: code = NotFound desc = could not find container \"cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84\": container with ID starting with cefbee156c4ed8a073cc243e59730071e65b72528ccb710584aa16003cc56a84 not found: ID does not exist"
Nov 03 22:08:33.787090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.787010    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 22:08:33.794731 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:33.794701    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c70fca-845478454c-8dcdx"]
Nov 03 22:08:34.004977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.004943    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 22:08:34.008847 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.008821    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c70fca-5cf8577756-tmzjn"]
Nov 03 22:08:34.099516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.099422    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main" containerID="cri-o://e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9" gracePeriod=45
Nov 03 22:08:34.365131 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.365059    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 22:08:34.438349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438314    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438372    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438469    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438497    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438545    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438742 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438613    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\" (UniqueName: \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\") pod \"45b1b737-1590-48aa-b291-5a2b465c779c\" (UID: \"45b1b737-1590-48aa-b291-5a2b465c779c\") "
Nov 03 22:08:34.438799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438735    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:34.438860 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438812    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:34.438860 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438837    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:34.438987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438962    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.438987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438975    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.438987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.438985    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.440523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.440497    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:34.440678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.440538    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:08:34.440845 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.440828    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m" pod "45b1b737-1590-48aa-b291-5a2b465c779c" (UID: "45b1b737-1590-48aa-b291-5a2b465c779c"). InnerVolumeSpecName "kube-api-access-qn85m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:34.496079 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.496047    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" path="/var/lib/kubelet/pods/7471ab21-70ed-4fb1-91c4-84caadfeb05a/volumes"
Nov 03 22:08:34.496699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.496613    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c811d2-bf9e-4076-b729-23324a24d3c8" path="/var/lib/kubelet/pods/f5c811d2-bf9e-4076-b729-23324a24d3c8/volumes"
Nov 03 22:08:34.539683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.539659    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn85m\" (UniqueName: \"kubernetes.io/projected/45b1b737-1590-48aa-b291-5a2b465c779c-kube-api-access-qn85m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.539683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.539684    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b1b737-1590-48aa-b291-5a2b465c779c-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.539842 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.539694    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b1b737-1590-48aa-b291-5a2b465c779c-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.642142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.642057    5533 generic.go:356] "Generic (PLEG): container finished" podID="45b1b737-1590-48aa-b291-5a2b465c779c" containerID="e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9" exitCode=0
Nov 03 22:08:34.642142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.642098    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"
Nov 03 22:08:34.642350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.642146    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" event={"ID":"45b1b737-1590-48aa-b291-5a2b465c779c","Type":"ContainerDied","Data":"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"}
Nov 03 22:08:34.642350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.642191    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h" event={"ID":"45b1b737-1590-48aa-b291-5a2b465c779c","Type":"ContainerDied","Data":"fce1358bdf345cd824970ab426cc41214eaf7febd08538437b0c47505211f587"}
Nov 03 22:08:34.642350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.642214    5533 scope.go:122] "RemoveContainer" containerID="e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"
Nov 03 22:08:34.653390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.653361    5533 scope.go:122] "RemoveContainer" containerID="e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"
Nov 03 22:08:34.653746 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:34.653708    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9\": container with ID starting with e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9 not found: ID does not exist" containerID="e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"
Nov 03 22:08:34.653843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.653754    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9"} err="failed to get container status \"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9\": rpc error: code = NotFound desc = could not find container \"e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9\": container with ID starting with e5e29e7cbb2d1fe16bc613856f40305a6c67fefdeb97ab98b2c9b0ee12d92ef9 not found: ID does not exist"
Nov 03 22:08:34.796442 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.796407    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:34.822788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.822758    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 22:08:34.832710 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.832683    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-099a5e/simulator-w2-099a5e-kserve-646b68d7c6-9777h"]
Nov 03 22:08:34.842169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.842145    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\") pod \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") "
Nov 03 22:08:34.842306 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.842181    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\" (UniqueName: \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\") pod \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") "
Nov 03 22:08:34.842306 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.842223    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\") pod \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\" (UID: \"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e\") "
Nov 03 22:08:34.842734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.842704    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca" pod "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" (UID: "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:34.845053 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.845027    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca" pod "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" (UID: "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:08:34.845162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.845029    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf" pod "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" (UID: "f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e"). InnerVolumeSpecName "kube-api-access-llgzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:34.943291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.943260    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.943291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.943287    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:34.943482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:34.943297    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgzf\" (UniqueName: \"kubernetes.io/projected/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e-kube-api-access-llgzf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:35.651988 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:35.651957    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"
Nov 03 22:08:35.651988 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:35.651971    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq" event={"ID":"f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e","Type":"ContainerDied","Data":"f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8"}
Nov 03 22:08:35.652507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:35.652020    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88ffdcd95f4d412e242745ea1211bf165f5ef22c28dd65177a93f695d3325b8"
Nov 03 22:08:36.492648 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:36.492603    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" path="/var/lib/kubelet/pods/45b1b737-1590-48aa-b291-5a2b465c779c/volumes"
Nov 03 22:08:37.572268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.572231    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" podUID="22c8fb54-cdba-4d31-872b-473b25eab3d9" containerName="payload-processing" containerID="cri-o://0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9" gracePeriod=25
Nov 03 22:08:37.682016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.681979    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" podUID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" containerName="payload-pre-processing" containerID="cri-o://3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f" gracePeriod=25
Nov 03 22:08:37.823085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.823025    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 22:08:37.869033 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.868999    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\" (UniqueName: \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\") pod \"22c8fb54-cdba-4d31-872b-473b25eab3d9\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") "
Nov 03 22:08:37.869173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.869160    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\") pod \"22c8fb54-cdba-4d31-872b-473b25eab3d9\" (UID: \"22c8fb54-cdba-4d31-872b-473b25eab3d9\") "
Nov 03 22:08:37.869564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.869541    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume" pod "22c8fb54-cdba-4d31-872b-473b25eab3d9" (UID: "22c8fb54-cdba-4d31-872b-473b25eab3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:37.871925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.871896    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6" pod "22c8fb54-cdba-4d31-872b-473b25eab3d9" (UID: "22c8fb54-cdba-4d31-872b-473b25eab3d9"). InnerVolumeSpecName "kube-api-access-c2zz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:37.918215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.918192    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 22:08:37.969942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.969915    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\" (UniqueName: \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\") pod \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") "
Nov 03 22:08:37.970090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.969967    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\") pod \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\" (UID: \"5dd5d6ba-284b-4d43-b965-d55c418c4e13\") "
Nov 03 22:08:37.970175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.970134    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c8fb54-cdba-4d31-872b-473b25eab3d9-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:37.970175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.970146    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zz6\" (UniqueName: \"kubernetes.io/projected/22c8fb54-cdba-4d31-872b-473b25eab3d9-kube-api-access-c2zz6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:37.970349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.970327    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume" pod "5dd5d6ba-284b-4d43-b965-d55c418c4e13" (UID: "5dd5d6ba-284b-4d43-b965-d55c418c4e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:37.971926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:37.971905    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b" pod "5dd5d6ba-284b-4d43-b965-d55c418c4e13" (UID: "5dd5d6ba-284b-4d43-b965-d55c418c4e13"). InnerVolumeSpecName "kube-api-access-dhb4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:38.071231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.071206    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhb4b\" (UniqueName: \"kubernetes.io/projected/5dd5d6ba-284b-4d43-b965-d55c418c4e13-kube-api-access-dhb4b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:38.071231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.071229    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5d6ba-284b-4d43-b965-d55c418c4e13-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:38.672745 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.672717    5533 generic.go:356] "Generic (PLEG): container finished" podID="22c8fb54-cdba-4d31-872b-473b25eab3d9" containerID="0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9" exitCode=0
Nov 03 22:08:38.673139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.672761    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"
Nov 03 22:08:38.673139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.672808    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" event={"ID":"22c8fb54-cdba-4d31-872b-473b25eab3d9","Type":"ContainerDied","Data":"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"}
Nov 03 22:08:38.673139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.672870    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75" event={"ID":"22c8fb54-cdba-4d31-872b-473b25eab3d9","Type":"ContainerDied","Data":"44ea7b6b3e8b868e619c4fa76596430dbeae473725f55e875b6d543cad019134"}
Nov 03 22:08:38.673139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.672906    5533 scope.go:122] "RemoveContainer" containerID="0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"
Nov 03 22:08:38.674327 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.674309    5533 generic.go:356] "Generic (PLEG): container finished" podID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" containerID="3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f" exitCode=0
Nov 03 22:08:38.674414 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.674345    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"
Nov 03 22:08:38.674414 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.674356    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" event={"ID":"5dd5d6ba-284b-4d43-b965-d55c418c4e13","Type":"ContainerDied","Data":"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"}
Nov 03 22:08:38.674414 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.674405    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g" event={"ID":"5dd5d6ba-284b-4d43-b965-d55c418c4e13","Type":"ContainerDied","Data":"ffd1ce9dfcae2ee686bb68494feaf2a9c4c4e4f8252e07f61497b214fd13c10c"}
Nov 03 22:08:38.682827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.682811    5533 scope.go:122] "RemoveContainer" containerID="0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"
Nov 03 22:08:38.683109 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:38.683088    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9\": container with ID starting with 0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9 not found: ID does not exist" containerID="0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"
Nov 03 22:08:38.683166 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.683115    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9"} err="failed to get container status \"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9\": rpc error: code = NotFound desc = could not find container \"0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9\": container with ID starting with 0b5fd615d7e2fa1acd4d17c8fbacf92e026436a3bc81afa7b9abc8d91e1c79a9 not found: ID does not exist"
Nov 03 22:08:38.683166 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.683132    5533 scope.go:122] "RemoveContainer" containerID="3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"
Nov 03 22:08:38.691385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.691368    5533 scope.go:122] "RemoveContainer" containerID="3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"
Nov 03 22:08:38.691660 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:38.691620    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f\": container with ID starting with 3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f not found: ID does not exist" containerID="3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"
Nov 03 22:08:38.691757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.691739    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f"} err="failed to get container status \"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f\": rpc error: code = NotFound desc = could not find container \"3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f\": container with ID starting with 3b6a1f49d9b133ff35f34f57e5c20e9afac46baa61016be80205ceef1dfc787f not found: ID does not exist"
Nov 03 22:08:38.828181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.828116    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 22:08:38.832173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.832148    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-099a5e-b4784ff6-s2b75"]
Nov 03 22:08:38.934106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.934083    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 22:08:38.940287 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:38.940267    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-099a5e-877d67599-zjh7g"]
Nov 03 22:08:40.493799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:40.493758    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c8fb54-cdba-4d31-872b-473b25eab3d9" path="/var/lib/kubelet/pods/22c8fb54-cdba-4d31-872b-473b25eab3d9/volumes"
Nov 03 22:08:40.494213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:40.494153    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" path="/var/lib/kubelet/pods/5dd5d6ba-284b-4d43-b965-d55c418c4e13/volumes"
Nov 03 22:08:48.292948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:48.292913    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 22:08:48.412134 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:48.412100    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 22:08:49.259989 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:49.259953    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:08:53.293427 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.293384    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" podUID="27255467-617e-4861-a359-b750691c1f84" containerName="payload-processing" containerID="cri-o://d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab" gracePeriod=25
Nov 03 22:08:53.413238 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.413191    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" podUID="a93e43ca-7512-480e-9126-fed363b62908" containerName="payload-pre-processing" containerID="cri-o://1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6" gracePeriod=25
Nov 03 22:08:53.539301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.539276    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 22:08:53.611217 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.611183    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\") pod \"27255467-617e-4861-a359-b750691c1f84\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") "
Nov 03 22:08:53.611383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.611237    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\" (UniqueName: \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\") pod \"27255467-617e-4861-a359-b750691c1f84\" (UID: \"27255467-617e-4861-a359-b750691c1f84\") "
Nov 03 22:08:53.611906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.611862    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume" pod "27255467-617e-4861-a359-b750691c1f84" (UID: "27255467-617e-4861-a359-b750691c1f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:53.614191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.614140    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl" pod "27255467-617e-4861-a359-b750691c1f84" (UID: "27255467-617e-4861-a359-b750691c1f84"). InnerVolumeSpecName "kube-api-access-276bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:53.657684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.657656    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 22:08:53.711982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.711951    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27255467-617e-4861-a359-b750691c1f84-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:53.711982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.711978    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276bl\" (UniqueName: \"kubernetes.io/projected/27255467-617e-4861-a359-b750691c1f84-kube-api-access-276bl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:53.785018 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.784992    5533 generic.go:356] "Generic (PLEG): container finished" podID="a93e43ca-7512-480e-9126-fed363b62908" containerID="1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6" exitCode=0
Nov 03 22:08:53.785136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.785042    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"
Nov 03 22:08:53.785136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.785080    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" event={"ID":"a93e43ca-7512-480e-9126-fed363b62908","Type":"ContainerDied","Data":"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"}
Nov 03 22:08:53.785224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.785136    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8" event={"ID":"a93e43ca-7512-480e-9126-fed363b62908","Type":"ContainerDied","Data":"fc4398a9eb7244fc09a589763caa706f8799153d39db57c769b182ab9848af44"}
Nov 03 22:08:53.785224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.785162    5533 scope.go:122] "RemoveContainer" containerID="1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"
Nov 03 22:08:53.786881 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.786861    5533 generic.go:356] "Generic (PLEG): container finished" podID="27255467-617e-4861-a359-b750691c1f84" containerID="d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab" exitCode=0
Nov 03 22:08:53.786985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.786904    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"
Nov 03 22:08:53.787061 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.786902    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" event={"ID":"27255467-617e-4861-a359-b750691c1f84","Type":"ContainerDied","Data":"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"}
Nov 03 22:08:53.787061 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.787042    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh" event={"ID":"27255467-617e-4861-a359-b750691c1f84","Type":"ContainerDied","Data":"e95b0db8fd404b7299634f9cd00a4171400e7c8caa815a0b150eceeb37d9193d"}
Nov 03 22:08:53.796118 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.796101    5533 scope.go:122] "RemoveContainer" containerID="1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"
Nov 03 22:08:53.796372 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:53.796349    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6\": container with ID starting with 1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6 not found: ID does not exist" containerID="1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"
Nov 03 22:08:53.796434 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.796378    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6"} err="failed to get container status \"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6\": rpc error: code = NotFound desc = could not find container \"1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6\": container with ID starting with 1bb6e81fc4d8a3d98e9ad09590c2e52c651eee0bd2db3f2b2eb74e7363e281c6 not found: ID does not exist"
Nov 03 22:08:53.796434 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.796396    5533 scope.go:122] "RemoveContainer" containerID="d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"
Nov 03 22:08:53.805260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.805241    5533 scope.go:122] "RemoveContainer" containerID="d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"
Nov 03 22:08:53.805513 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:08:53.805492    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab\": container with ID starting with d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab not found: ID does not exist" containerID="d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"
Nov 03 22:08:53.805604 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.805515    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab"} err="failed to get container status \"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab\": rpc error: code = NotFound desc = could not find container \"d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab\": container with ID starting with d0bdaa0c7e64bf953121cf7599c1790d63d295d291cd2546599fa181e8114eab not found: ID does not exist"
Nov 03 22:08:53.812807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.812789    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\" (UniqueName: \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\") pod \"a93e43ca-7512-480e-9126-fed363b62908\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") "
Nov 03 22:08:53.812934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.812920    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\") pod \"a93e43ca-7512-480e-9126-fed363b62908\" (UID: \"a93e43ca-7512-480e-9126-fed363b62908\") "
Nov 03 22:08:53.813245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.813226    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume" pod "a93e43ca-7512-480e-9126-fed363b62908" (UID: "a93e43ca-7512-480e-9126-fed363b62908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:53.814747 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.814727    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr" pod "a93e43ca-7512-480e-9126-fed363b62908" (UID: "a93e43ca-7512-480e-9126-fed363b62908"). InnerVolumeSpecName "kube-api-access-2wtjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:53.913567 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.913519    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93e43ca-7512-480e-9126-fed363b62908-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:53.913567 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.913541    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wtjr\" (UniqueName: \"kubernetes.io/projected/a93e43ca-7512-480e-9126-fed363b62908-kube-api-access-2wtjr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:08:53.939584 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.939561    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 22:08:53.943777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:53.943756    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a1c634-7fc59fb67d-969dh"]
Nov 03 22:08:54.199855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:54.199827    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 22:08:54.203839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:54.203813    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a1c634-745f7557ff-tqgt8"]
Nov 03 22:08:54.491683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:54.491616    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27255467-617e-4861-a359-b750691c1f84" path="/var/lib/kubelet/pods/27255467-617e-4861-a359-b750691c1f84/volumes"
Nov 03 22:08:54.492007 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:08:54.491992    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93e43ca-7512-480e-9126-fed363b62908" path="/var/lib/kubelet/pods/a93e43ca-7512-480e-9126-fed363b62908/volumes"
Nov 03 22:09:00.045453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:00.045413    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"]
Nov 03 22:09:00.091923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:00.091892    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca14dd43-ssk7w"]
Nov 03 22:09:00.492342 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:00.492304    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd809bef-5551-416e-a714-72347a4ad232" path="/var/lib/kubelet/pods/fd809bef-5551-416e-a714-72347a4ad232/volumes"
Nov 03 22:09:02.578776 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:02.578736    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 22:09:02.579276 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:02.578961    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy" containerID="cri-o://6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65" gracePeriod=30
Nov 03 22:09:02.581400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:02.581372    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:09:03.834141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:03.834103    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 22:09:03.834546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:03.834301    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy" containerID="cri-o://288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18" gracePeriod=30
Nov 03 22:09:04.891034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.890996    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"]
Nov 03 22:09:04.891554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891529    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" containerName="payload-pre-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891558    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" containerName="payload-pre-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891577    5533 cpu_manager.go:414] "containerMap: removing container" podUID="27255467-617e-4861-a359-b750691c1f84" containerName="payload-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891585    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27255467-617e-4861-a359-b750691c1f84" containerName="payload-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891605    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" containerName="revoke-keys"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891613    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" containerName="revoke-keys"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891648    5533 cpu_manager.go:414] "containerMap: removing container" podUID="22c8fb54-cdba-4d31-872b-473b25eab3d9" containerName="payload-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891657    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c8fb54-cdba-4d31-872b-473b25eab3d9" containerName="payload-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891673    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" containerName="payload-processing"
Nov 03 22:09:04.891684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891681    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" containerName="payload-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891699    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a93e43ca-7512-480e-9126-fed363b62908" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891707    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93e43ca-7512-480e-9126-fed363b62908" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891725    5533 cpu_manager.go:414] "containerMap: removing container" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891733    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891743    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f5c811d2-bf9e-4076-b729-23324a24d3c8" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891751    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c811d2-bf9e-4076-b729-23324a24d3c8" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891851    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd5d6ba-284b-4d43-b965-d55c418c4e13" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891867    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="27255467-617e-4861-a359-b750691c1f84" containerName="payload-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891877    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" containerName="revoke-keys"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891889    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c8fb54-cdba-4d31-872b-473b25eab3d9" containerName="payload-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891899    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7471ab21-70ed-4fb1-91c4-84caadfeb05a" containerName="payload-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891909    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93e43ca-7512-480e-9126-fed363b62908" containerName="payload-pre-processing"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891919    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b1b737-1590-48aa-b291-5a2b465c779c" containerName="main"
Nov 03 22:09:04.892171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.891927    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c811d2-bf9e-4076-b729-23324a24d3c8" containerName="payload-pre-processing"
Nov 03 22:09:04.895337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.895315    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:04.949726 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.949698    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"]
Nov 03 22:09:04.993421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.993398    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:09:04.993538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:04.993437    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:09:05.006342 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.006314    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.006440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.006360    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.006519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.006499    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscxw\" (UniqueName: \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.106833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.106805    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.106931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.106849    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.106972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.106940    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscxw\" (UniqueName: \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.107544 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.107516    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.109336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.109316    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.164209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.164157    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscxw\" (UniqueName: \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\") pod \"maas-api-revoke-sub-9061f323-nhbgs\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.207601 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.207577    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:05.572788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.572756    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"]
Nov 03 22:09:05.574592 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:09:05.574561    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f293a7_5b5e_4585_bdbb_f51d3dbc0569.slice/crio-3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca WatchSource:0}: Error finding container 3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca: Status 404 returned error can't find the container with id 3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca
Nov 03 22:09:05.861733 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.861673    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs" event={"ID":"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569","Type":"ContainerStarted","Data":"203204d1b60e39f8d2dc4437eb681c9e4dd09e59f1f8a0160cbce62a7c5d9c76"}
Nov 03 22:09:05.861733 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.861726    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs" event={"ID":"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569","Type":"ContainerStarted","Data":"3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca"}
Nov 03 22:09:05.970531 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:05.970486    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs" podStartSLOduration=1.970473899 podStartE2EDuration="1.970473899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:05.969933346 +0000 UTC m=+4847.601246824" watchObservedRunningTime="2026-11-03 22:09:05.970473899 +0000 UTC m=+4847.601787375"
Nov 03 22:09:06.869133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:06.869041    5533 generic.go:356] "Generic (PLEG): container finished" podID="f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" containerID="203204d1b60e39f8d2dc4437eb681c9e4dd09e59f1f8a0160cbce62a7c5d9c76" exitCode=0
Nov 03 22:09:06.869293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:06.869130    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs" event={"ID":"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569","Type":"ContainerDied","Data":"203204d1b60e39f8d2dc4437eb681c9e4dd09e59f1f8a0160cbce62a7c5d9c76"}
Nov 03 22:09:07.824717 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.824694    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 22:09:07.876222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.876172    5533 generic.go:356] "Generic (PLEG): container finished" podID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerID="6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65" exitCode=0
Nov 03 22:09:07.876222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.876215    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"
Nov 03 22:09:07.876413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.876252    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerDied","Data":"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"}
Nov 03 22:09:07.876413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.876313    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n" event={"ID":"49b4b7fb-21d4-48ee-8d02-aa888260eff0","Type":"ContainerDied","Data":"39599aa8ca2e4d067e0ec8a7055589d6816fcdfe9d73e1644aa3c7933548a869"}
Nov 03 22:09:07.876413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.876344    5533 scope.go:122] "RemoveContainer" containerID="6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"
Nov 03 22:09:07.885652 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.885617    5533 scope.go:122] "RemoveContainer" containerID="e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"
Nov 03 22:09:07.898165 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.898146    5533 scope.go:122] "RemoveContainer" containerID="6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"
Nov 03 22:09:07.898490 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:07.898468    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65\": container with ID starting with 6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65 not found: ID does not exist" containerID="6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"
Nov 03 22:09:07.898558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.898499    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65"} err="failed to get container status \"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65\": rpc error: code = NotFound desc = could not find container \"6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65\": container with ID starting with 6df00c3f59fd9ee2512640d532c14ff7f939581167edba03faf92d5af464ba65 not found: ID does not exist"
Nov 03 22:09:07.898558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.898517    5533 scope.go:122] "RemoveContainer" containerID="e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"
Nov 03 22:09:07.898817 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:07.898801    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567\": container with ID starting with e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567 not found: ID does not exist" containerID="e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"
Nov 03 22:09:07.898868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.898824    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567"} err="failed to get container status \"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567\": rpc error: code = NotFound desc = could not find container \"e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567\": container with ID starting with e7ac2dec08aed9a4fb3363321d5ffafee9b306d39fa0f1f97226ff3c4d2f6567 not found: ID does not exist"
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931006    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931052    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931115    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931153    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931224    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931292    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931346    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931389    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.931416    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\") pod \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\" (UID: \"49b4b7fb-21d4-48ee-8d02-aa888260eff0\") "
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.932505    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.933139    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.933329    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.933799    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:07.936666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.934129    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:07.948941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.948916    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:07.949478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.949439    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "kube-api-access-9l9fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:07.949704 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.949681    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:09:07.949838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.949815    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token" pod "49b4b7fb-21d4-48ee-8d02-aa888260eff0" (UID: "49b4b7fb-21d4-48ee-8d02-aa888260eff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:07.995358 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:07.995332    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:08.032215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032191    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032214    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032224    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032233    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032241    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032252    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032261    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032269    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9fp\" (UniqueName: \"kubernetes.io/projected/49b4b7fb-21d4-48ee-8d02-aa888260eff0-kube-api-access-9l9fp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.032353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.032278    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b4b7fb-21d4-48ee-8d02-aa888260eff0-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.133420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.133339    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\" (UniqueName: \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\") pod \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") "
Nov 03 22:09:08.133560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.133422    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\") pod \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") "
Nov 03 22:09:08.133560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.133486    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\") pod \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\" (UID: \"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569\") "
Nov 03 22:09:08.133849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.133824    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca" pod "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" (UID: "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:08.133946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.133895    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.135525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.135505    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca" pod "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" (UID: "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:09:08.135603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.135585    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw" pod "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" (UID: "f2f293a7-5b5e-4585-bdbb-f51d3dbc0569"). InnerVolumeSpecName "kube-api-access-fscxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:08.234402 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.234376    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fscxw\" (UniqueName: \"kubernetes.io/projected/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-kube-api-access-fscxw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.234402 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.234399    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:08.292154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.292127    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 22:09:08.295845 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.295823    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-099a5e-openshift-default-7bbb9c8b87-k5q7n"]
Nov 03 22:09:08.493113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.493087    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" path="/var/lib/kubelet/pods/49b4b7fb-21d4-48ee-8d02-aa888260eff0/volumes"
Nov 03 22:09:08.883743 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.883704    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"
Nov 03 22:09:08.884071 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.883707    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs" event={"ID":"f2f293a7-5b5e-4585-bdbb-f51d3dbc0569","Type":"ContainerDied","Data":"3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca"}
Nov 03 22:09:08.884071 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:08.883895    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b400d89e1582bd5b510c70fe17da788a7c5dd4be981871cec3d906717bb10ca"
Nov 03 22:09:09.078849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.078826    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 22:09:09.242459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242429    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242475    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242495    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242516    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242602 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242539    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242729    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242763    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242800    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.242875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.242819    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\") pod \"0fe62722-5163-49cb-9d8d-2704bace267e\" (UID: \"0fe62722-5163-49cb-9d8d-2704bace267e\") "
Nov 03 22:09:09.243431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.243273    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:09.243431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.243317    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:09.243431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.243337    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:09.243431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.243392    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:09.243731 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.243498    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:09.245127 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.245098    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:09.245252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.245128    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:09.245252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.245194    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:09:09.245380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.245345    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr" pod "0fe62722-5163-49cb-9d8d-2704bace267e" (UID: "0fe62722-5163-49cb-9d8d-2704bace267e"). InnerVolumeSpecName "kube-api-access-kj6mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:09.290863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.290836    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 22:09:09.343854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343828    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343852    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj6mr\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-kube-api-access-kj6mr\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343863    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343872    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe62722-5163-49cb-9d8d-2704bace267e-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343880    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe62722-5163-49cb-9d8d-2704bace267e-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343888    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343897    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343905    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe62722-5163-49cb-9d8d-2704bace267e-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.343994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.343912    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe62722-5163-49cb-9d8d-2704bace267e-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:09.892584 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.892551    5533 generic.go:356] "Generic (PLEG): container finished" podID="0fe62722-5163-49cb-9d8d-2704bace267e" containerID="288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18" exitCode=0
Nov 03 22:09:09.893085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.892605    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"
Nov 03 22:09:09.893085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.892652    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerDied","Data":"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"}
Nov 03 22:09:09.893085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.892710    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx" event={"ID":"0fe62722-5163-49cb-9d8d-2704bace267e","Type":"ContainerDied","Data":"267ecc30ff61530d3760d3f2cdbd1aa1c64efe45cc668b49363875ab557f140b"}
Nov 03 22:09:09.893085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.892732    5533 scope.go:122] "RemoveContainer" containerID="288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"
Nov 03 22:09:09.903278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.903263    5533 scope.go:122] "RemoveContainer" containerID="7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"
Nov 03 22:09:09.911630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.911614    5533 scope.go:122] "RemoveContainer" containerID="288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"
Nov 03 22:09:09.911927 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:09.911907    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18\": container with ID starting with 288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18 not found: ID does not exist" containerID="288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"
Nov 03 22:09:09.911978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.911935    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18"} err="failed to get container status \"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18\": rpc error: code = NotFound desc = could not find container \"288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18\": container with ID starting with 288f205ee442702a4202aa6b24a23218ee73d8ad9695a4448b877b836cdf7f18 not found: ID does not exist"
Nov 03 22:09:09.911978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.911951    5533 scope.go:122] "RemoveContainer" containerID="7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"
Nov 03 22:09:09.912173 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:09.912154    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185\": container with ID starting with 7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185 not found: ID does not exist" containerID="7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"
Nov 03 22:09:09.912249 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:09.912182    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185"} err="failed to get container status \"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185\": rpc error: code = NotFound desc = could not find container \"7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185\": container with ID starting with 7beab5cadf55a2cb2151449d2f1a3710bc277e1d6981ff351d976b391b085185 not found: ID does not exist"
Nov 03 22:09:10.050614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:10.050582    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 22:09:10.096885 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:10.096863    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a1c634-openshift-default-789bbd4977-94chx"]
Nov 03 22:09:10.491729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:10.491702    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" path="/var/lib/kubelet/pods/0fe62722-5163-49cb-9d8d-2704bace267e/volumes"
Nov 03 22:09:19.046104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:19.046065    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"]
Nov 03 22:09:19.092721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:19.092693    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79603805-tv4lk"]
Nov 03 22:09:19.260580 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:19.260544    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:09:19.314455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:19.314389    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:09:20.491748 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:20.491711    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3783ee3b-5f53-4705-b690-20030433d97f" path="/var/lib/kubelet/pods/3783ee3b-5f53-4705-b690-20030433d97f/volumes"
Nov 03 22:09:21.690883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.690851    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"]
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691189    5533 cpu_manager.go:414] "containerMap: removing container" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691200    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691226    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691231    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691243    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" containerName="revoke-keys"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691248    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" containerName="revoke-keys"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691256    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.691258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691261    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.691512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691318    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:09:21.691512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691329    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:09:21.691512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691336    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.691512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691342    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" containerName="revoke-keys"
Nov 03 22:09:21.691512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.691349    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe62722-5163-49cb-9d8d-2704bace267e" containerName="istio-proxy"
Nov 03 22:09:21.694461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.694445    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.741901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.741872    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.742029 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.741931    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztx6\" (UniqueName: \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.742108 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.742032    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.743750 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.743737    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:09:21.790294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.790267    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hp2vv\""
Nov 03 22:09:21.792831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.792809    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"]
Nov 03 22:09:21.842968 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.842940    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.843096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.843033    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztx6\" (UniqueName: \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.843096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.843088    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.843685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.843665    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.845185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.845156    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:21.900741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:21.900719    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztx6\" (UniqueName: \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\") pod \"maas-api-revoke-sub-ad411417-6vtqw\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:22.007138 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.007072    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:22.017247 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.017223    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 22:09:22.114392 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.114360    5533 scope.go:122] "RemoveContainer" containerID="d45f1aad7520498531ce738a2a7e226d54c3621123ccf67c1a9f5b5246827ba1"
Nov 03 22:09:22.124555 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.124534    5533 scope.go:122] "RemoveContainer" containerID="6d168117208441456724510ae51a7e0880d7ec2f2ddf3d522dc4afe8203b1f0a"
Nov 03 22:09:22.132419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.132395    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 22:09:22.373260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.373230    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"]
Nov 03 22:09:22.374835 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:09:22.374807    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b88ad4_4816_4515_9aa2_ba2e3625b760.slice/crio-a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26 WatchSource:0}: Error finding container a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26: Status 404 returned error can't find the container with id a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26
Nov 03 22:09:22.971037 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.971003    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw" event={"ID":"96b88ad4-4816-4515-9aa2-ba2e3625b760","Type":"ContainerStarted","Data":"36ea46baf9bfec8fffe53d6079acb767d7c2b2ad6a0a75203322e2860c9f0e9c"}
Nov 03 22:09:22.971475 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:22.971060    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw" event={"ID":"96b88ad4-4816-4515-9aa2-ba2e3625b760","Type":"ContainerStarted","Data":"a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26"}
Nov 03 22:09:23.080929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:23.080877    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw" podStartSLOduration=2.080862372 podStartE2EDuration="2.080862372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:23.079373518 +0000 UTC m=+4864.710687034" watchObservedRunningTime="2026-11-03 22:09:23.080862372 +0000 UTC m=+4864.712175849"
Nov 03 22:09:23.978884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:23.978850    5533 generic.go:356] "Generic (PLEG): container finished" podID="96b88ad4-4816-4515-9aa2-ba2e3625b760" containerID="36ea46baf9bfec8fffe53d6079acb767d7c2b2ad6a0a75203322e2860c9f0e9c" exitCode=0
Nov 03 22:09:23.979281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:23.978931    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw" event={"ID":"96b88ad4-4816-4515-9aa2-ba2e3625b760","Type":"ContainerDied","Data":"36ea46baf9bfec8fffe53d6079acb767d7c2b2ad6a0a75203322e2860c9f0e9c"}
Nov 03 22:09:24.310331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.310251    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main" containerID="cri-o://4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8" gracePeriod=45
Nov 03 22:09:24.555831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.555804    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 22:09:24.561746 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561691    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.561746 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561739    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.561931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561777    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.561931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561820    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.561931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561864    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.561931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561896    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\" (UniqueName: \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\") pod \"95fad564-be5b-462d-a0a3-3a48c7826f63\" (UID: \"95fad564-be5b-462d-a0a3-3a48c7826f63\") "
Nov 03 22:09:24.562192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.561970    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:24.562192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.562075    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:24.562313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.562198    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:24.562313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.562260    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.562313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.562295    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.564301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.564270    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "kube-api-access-4cjkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:24.564652 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.564610    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:09:24.564652 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.564623    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm" pod "95fad564-be5b-462d-a0a3-3a48c7826f63" (UID: "95fad564-be5b-462d-a0a3-3a48c7826f63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:09:24.662902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.662869    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.662902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.662901    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95fad564-be5b-462d-a0a3-3a48c7826f63-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.663076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.662917    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjkq\" (UniqueName: \"kubernetes.io/projected/95fad564-be5b-462d-a0a3-3a48c7826f63-kube-api-access-4cjkq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.663076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.662931    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fad564-be5b-462d-a0a3-3a48c7826f63-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:24.987945 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.987910    5533 generic.go:356] "Generic (PLEG): container finished" podID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerID="4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8" exitCode=0
Nov 03 22:09:24.988415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.987962    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"
Nov 03 22:09:24.988415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.988009    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" event={"ID":"95fad564-be5b-462d-a0a3-3a48c7826f63","Type":"ContainerDied","Data":"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"}
Nov 03 22:09:24.988415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.988062    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2" event={"ID":"95fad564-be5b-462d-a0a3-3a48c7826f63","Type":"ContainerDied","Data":"4e271a6d75e42654f89c3a6e2ff96a97d4014353546b9e1c228dcd79c33bda46"}
Nov 03 22:09:24.988415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.988083    5533 scope.go:122] "RemoveContainer" containerID="4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"
Nov 03 22:09:24.999227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.999206    5533 scope.go:122] "RemoveContainer" containerID="4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"
Nov 03 22:09:24.999491 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:24.999466    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8\": container with ID starting with 4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8 not found: ID does not exist" containerID="4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"
Nov 03 22:09:24.999553 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:24.999503    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8"} err="failed to get container status \"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8\": rpc error: code = NotFound desc = could not find container \"4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8\": container with ID starting with 4293df5f465e47226dae6322b4116f30d3e39b6d250197c9a393be4e65616bb8 not found: ID does not exist"
Nov 03 22:09:25.121681 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.121657    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:25.143569 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.143545    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 22:09:25.149056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.149029    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e84357/premium-w4-e84357-kserve-54d766fdcd-24tr2"]
Nov 03 22:09:25.166672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.166626    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\") pod \"96b88ad4-4816-4515-9aa2-ba2e3625b760\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") "
Nov 03 22:09:25.166801 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.166725    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\" (UniqueName: \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\") pod \"96b88ad4-4816-4515-9aa2-ba2e3625b760\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") "
Nov 03 22:09:25.166801 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.166762    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\") pod \"96b88ad4-4816-4515-9aa2-ba2e3625b760\" (UID: \"96b88ad4-4816-4515-9aa2-ba2e3625b760\") "
Nov 03 22:09:25.166934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.166913    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca" pod "96b88ad4-4816-4515-9aa2-ba2e3625b760" (UID: "96b88ad4-4816-4515-9aa2-ba2e3625b760"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:25.167038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.167017    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:25.168735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.168707    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca" pod "96b88ad4-4816-4515-9aa2-ba2e3625b760" (UID: "96b88ad4-4816-4515-9aa2-ba2e3625b760"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:09:25.168837 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.168765    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6" pod "96b88ad4-4816-4515-9aa2-ba2e3625b760" (UID: "96b88ad4-4816-4515-9aa2-ba2e3625b760"). InnerVolumeSpecName "kube-api-access-dztx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:25.267426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.267370    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztx6\" (UniqueName: \"kubernetes.io/projected/96b88ad4-4816-4515-9aa2-ba2e3625b760-kube-api-access-dztx6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:25.267426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.267392    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b88ad4-4816-4515-9aa2-ba2e3625b760-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:25.994869 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.994840    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"
Nov 03 22:09:25.995283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.994870    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw" event={"ID":"96b88ad4-4816-4515-9aa2-ba2e3625b760","Type":"ContainerDied","Data":"a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26"}
Nov 03 22:09:25.995283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:25.994920    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a917edbb4c1d5d0f423a9e2f65337092155749225e398026e3ea65737d3b3d26"
Nov 03 22:09:26.492570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:26.492529    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" path="/var/lib/kubelet/pods/95fad564-be5b-462d-a0a3-3a48c7826f63/volumes"
Nov 03 22:09:27.018064 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:27.018026    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing" containerID="cri-o://329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc" gracePeriod=25
Nov 03 22:09:27.133181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:27.133130    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing" containerID="cri-o://fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b" gracePeriod=25
Nov 03 22:09:28.180885 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:28.180846    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:09:28.181251 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:28.180913    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:09:36.662948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:36.662916    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:09:38.180409 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:38.180366    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:09:38.180840 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:38.180376    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:09:44.259692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:44.259657    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:09:47.257785 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.257762    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 22:09:47.331712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.331686    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\" (UniqueName: \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\") pod \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") "
Nov 03 22:09:47.331849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.331725    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\") pod \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\" (UID: \"502ae4a2-d25a-4106-b927-e6041c4d0dbc\") "
Nov 03 22:09:47.332228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.332200    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume" pod "502ae4a2-d25a-4106-b927-e6041c4d0dbc" (UID: "502ae4a2-d25a-4106-b927-e6041c4d0dbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:47.333830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.333804    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j" pod "502ae4a2-d25a-4106-b927-e6041c4d0dbc" (UID: "502ae4a2-d25a-4106-b927-e6041c4d0dbc"). InnerVolumeSpecName "kube-api-access-w7t7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:47.364685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.364665    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 22:09:47.432207 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.432167    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\" (UniqueName: \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\") pod \"f19c9055-2190-4be0-b961-02d4cf3db07c\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") "
Nov 03 22:09:47.432388 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.432282    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\") pod \"f19c9055-2190-4be0-b961-02d4cf3db07c\" (UID: \"f19c9055-2190-4be0-b961-02d4cf3db07c\") "
Nov 03 22:09:47.432476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.432421    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7t7j\" (UniqueName: \"kubernetes.io/projected/502ae4a2-d25a-4106-b927-e6041c4d0dbc-kube-api-access-w7t7j\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:47.432476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.432433    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502ae4a2-d25a-4106-b927-e6041c4d0dbc-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:47.432732 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.432705    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume" pod "f19c9055-2190-4be0-b961-02d4cf3db07c" (UID: "f19c9055-2190-4be0-b961-02d4cf3db07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:47.434571 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.434551    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7" pod "f19c9055-2190-4be0-b961-02d4cf3db07c" (UID: "f19c9055-2190-4be0-b961-02d4cf3db07c"). InnerVolumeSpecName "kube-api-access-kbwq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:47.533202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.533165    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwq7\" (UniqueName: \"kubernetes.io/projected/f19c9055-2190-4be0-b961-02d4cf3db07c-kube-api-access-kbwq7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:47.533202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:47.533199    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c9055-2190-4be0-b961-02d4cf3db07c-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:09:48.120447 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.120414    5533 generic.go:356] "Generic (PLEG): container finished" podID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerID="329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc" exitCode=0
Nov 03 22:09:48.120628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.120478    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"
Nov 03 22:09:48.120628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.120505    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" event={"ID":"502ae4a2-d25a-4106-b927-e6041c4d0dbc","Type":"ContainerDied","Data":"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"}
Nov 03 22:09:48.120628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.120582    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4" event={"ID":"502ae4a2-d25a-4106-b927-e6041c4d0dbc","Type":"ContainerDied","Data":"bcd779c6ca8fe8b8f7b9b900a987b043e902cfc8b8ab5970ceb897105a2c79b4"}
Nov 03 22:09:48.120628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.120610    5533 scope.go:122] "RemoveContainer" containerID="329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"
Nov 03 22:09:48.122154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.122130    5533 generic.go:356] "Generic (PLEG): container finished" podID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerID="fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b" exitCode=0
Nov 03 22:09:48.122270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.122163    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" event={"ID":"f19c9055-2190-4be0-b961-02d4cf3db07c","Type":"ContainerDied","Data":"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"}
Nov 03 22:09:48.122270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.122210    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"
Nov 03 22:09:48.122270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.122223    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f" event={"ID":"f19c9055-2190-4be0-b961-02d4cf3db07c","Type":"ContainerDied","Data":"e0f6374fb70d8ec7dab04f47569846d1794828b1b08a3cc44344e4c21f42db24"}
Nov 03 22:09:48.131561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.131402    5533 scope.go:122] "RemoveContainer" containerID="329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"
Nov 03 22:09:48.131719 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:48.131702    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc\": container with ID starting with 329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc not found: ID does not exist" containerID="329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"
Nov 03 22:09:48.131761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.131726    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc"} err="failed to get container status \"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc\": rpc error: code = NotFound desc = could not find container \"329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc\": container with ID starting with 329ef57e298cb5fd00615d5b6eb15f073237aef1b19d041b49a05a4f471e1dfc not found: ID does not exist"
Nov 03 22:09:48.131761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.131743    5533 scope.go:122] "RemoveContainer" containerID="fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"
Nov 03 22:09:48.139517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.139499    5533 scope.go:122] "RemoveContainer" containerID="fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"
Nov 03 22:09:48.139818 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:48.139799    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b\": container with ID starting with fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b not found: ID does not exist" containerID="fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"
Nov 03 22:09:48.139885 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.139827    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b"} err="failed to get container status \"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b\": rpc error: code = NotFound desc = could not find container \"fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b\": container with ID starting with fbf19d517a0ee1869e0609b8060cbe6be5dffea22f648e4dca9a9ebe31ab385b not found: ID does not exist"
Nov 03 22:09:48.278371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.278345    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 22:09:48.325185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.325135    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e84357-759966f7bc-2nv8f"]
Nov 03 22:09:48.386253 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.386226    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 22:09:48.388935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.388915    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e84357-8bdc47866-qcjw4"]
Nov 03 22:09:48.492371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.492348    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" path="/var/lib/kubelet/pods/502ae4a2-d25a-4106-b927-e6041c4d0dbc/volumes"
Nov 03 22:09:48.492722 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:48.492709    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" path="/var/lib/kubelet/pods/f19c9055-2190-4be0-b961-02d4cf3db07c/volumes"
Nov 03 22:09:49.260172 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:09:49.260131    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:09:52.692113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:52.692086    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csxht_8fac1941-f4d4-4e7d-ad78-a6c93434daf3/kuadrant-console-plugin/0.log"
Nov 03 22:09:53.909826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:53.909743    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gn82t_e93933ed-7935-46bf-8a67-258fda1d0fd6/manager/0.log"
Nov 03 22:09:54.181444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:54.181413    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kcbgn_18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3/limitador/0.log"
Nov 03 22:09:54.416685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:09:54.416654    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ngx24_128dab95-58a5-4fc4-bc64-21b427628fc2/manager/0.log"
Nov 03 22:10:03.146210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146173    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146482    5533 cpu_manager.go:414] "containerMap: removing container" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146493    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b4b7fb-21d4-48ee-8d02-aa888260eff0" containerName="istio-proxy"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146506    5533 cpu_manager.go:414] "containerMap: removing container" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146511    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146529    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146535    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146542    5533 cpu_manager.go:414] "containerMap: removing container" podUID="96b88ad4-4816-4515-9aa2-ba2e3625b760" containerName="revoke-keys"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146546    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b88ad4-4816-4515-9aa2-ba2e3625b760" containerName="revoke-keys"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146556    5533 cpu_manager.go:414] "containerMap: removing container" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146560    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146618    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b88ad4-4816-4515-9aa2-ba2e3625b760" containerName="revoke-keys"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146627    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="502ae4a2-d25a-4106-b927-e6041c4d0dbc" containerName="payload-processing"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146658    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fad564-be5b-462d-a0a3-3a48c7826f63" containerName="main"
Nov 03 22:10:03.146701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.146674    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19c9055-2190-4be0-b961-02d4cf3db07c" containerName="payload-pre-processing"
Nov 03 22:10:03.149615 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.149591    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.199148 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.199113    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4edfb8c0-openshift-default-dockercfg-82r85\""
Nov 03 22:10:03.249534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.249494    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:10:03.254853 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.254820    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.254987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.254854    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.254987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.254876    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cc6\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.254987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.254958    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.255137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.254987    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.255137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.255082    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.255230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.255152    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.255230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.255207    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.255340 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.255234    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356380    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356421    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356531    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356574    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356706    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356756    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cc6\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356810    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.356873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356839    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356916    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356923    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.356985    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.357091    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357268 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.357210    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.357534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.357330    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.358866 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.358842    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.359489 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.359467    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.414521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.414462    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.414521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.414494    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cc6\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\") pod \"e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") " pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.461797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.461763    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:03.881963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.881939    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:10:03.883486 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:10:03.883460    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169d8aca_c814_4057_88f5_d0117bea9952.slice/crio-01c8902e94e91b1771358782dbe18bc7b53973dec7614f477f0465fc10c5b77b WatchSource:0}: Error finding container 01c8902e94e91b1771358782dbe18bc7b53973dec7614f477f0465fc10c5b77b: Status 404 returned error can't find the container with id 01c8902e94e91b1771358782dbe18bc7b53973dec7614f477f0465fc10c5b77b
Nov 03 22:10:03.885615 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.885585    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:03.885708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.885684    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:03.885757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:03.885714    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:04.045311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.045280    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"]
Nov 03 22:10:04.048052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.048028    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d2bf94-gzk26"]
Nov 03 22:10:04.217809 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.217762    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerStarted","Data":"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"}
Nov 03 22:10:04.218291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.217828    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerStarted","Data":"01c8902e94e91b1771358782dbe18bc7b53973dec7614f477f0465fc10c5b77b"}
Nov 03 22:10:04.332542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.332486    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podStartSLOduration=1.332467535 podStartE2EDuration="1.332467535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:04.330769062 +0000 UTC m=+4905.962082539" watchObservedRunningTime="2026-11-03 22:10:04.332467535 +0000 UTC m=+4905.963781014"
Nov 03 22:10:04.462886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.462858    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:04.467981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.467928    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:04.491448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:04.491413    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1247b40b-55ca-432a-83a5-f66793bc6029" path="/var/lib/kubelet/pods/1247b40b-55ca-432a-83a5-f66793bc6029/volumes"
Nov 03 22:10:05.223528 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:05.223498    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:05.224334 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:05.224313    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:10.067194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.067157    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:10:10.071041 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.071022    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.110839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.110813    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.110954 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.110845    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkzh\" (UniqueName: \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.123202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.123178    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4edfb8c0\""
Nov 03 22:10:10.172911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.172880    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:10:10.211554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.211514    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.211554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.211554    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkzh\" (UniqueName: \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.211793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.211564    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:10:10.212297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.212275    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.215240 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.215221    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.216524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.216500    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4edfb8c0-dockercfg-cs5xk\""
Nov 03 22:10:10.269907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.269880    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkzh\" (UniqueName: \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\") pod \"payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.270036 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.269972    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:10:10.312572 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.312540    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvs4\" (UniqueName: \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.312715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.312620    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.384245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.384190    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:10.413404 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.413381    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvs4\" (UniqueName: \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.413511 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.413437    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.414137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.414120    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.471255 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.471223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvs4\" (UniqueName: \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\") pod \"payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") " pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.531564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.531537    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:10.748921 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.748889    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:10:10.749754 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:10:10.749726    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a383a7_dfb4_48d2_8d74_1b1a87903e84.slice/crio-d500785a407f9994f15a5ce867c6e9bf4a939b282aa7b14e5f1d9268ddf8a924 WatchSource:0}: Error finding container d500785a407f9994f15a5ce867c6e9bf4a939b282aa7b14e5f1d9268ddf8a924: Status 404 returned error can't find the container with id d500785a407f9994f15a5ce867c6e9bf4a939b282aa7b14e5f1d9268ddf8a924
Nov 03 22:10:10.898778 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:10.898750    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:10:10.899723 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:10:10.899698    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b93452_e330_41b4_bd29_dcec97108bf9.slice/crio-b86e813f805c576d5347dc82729b2936b87c523d545fa9f0538cbbef995cd417 WatchSource:0}: Error finding container b86e813f805c576d5347dc82729b2936b87c523d545fa9f0538cbbef995cd417: Status 404 returned error can't find the container with id b86e813f805c576d5347dc82729b2936b87c523d545fa9f0538cbbef995cd417
Nov 03 22:10:11.259175 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.259141    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" event={"ID":"43b93452-e330-41b4-bd29-dcec97108bf9","Type":"ContainerStarted","Data":"36b174b92bf828edb24ce7a79c1057745dc48e0e34f3e2e9018fcd5ef5624d19"}
Nov 03 22:10:11.259585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.259200    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" event={"ID":"43b93452-e330-41b4-bd29-dcec97108bf9","Type":"ContainerStarted","Data":"b86e813f805c576d5347dc82729b2936b87c523d545fa9f0538cbbef995cd417"}
Nov 03 22:10:11.259585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.259342    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:11.260775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.260754    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" event={"ID":"e1a383a7-dfb4-48d2-8d74-1b1a87903e84","Type":"ContainerStarted","Data":"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"}
Nov 03 22:10:11.260876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.260795    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:11.260876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.260810    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" event={"ID":"e1a383a7-dfb4-48d2-8d74-1b1a87903e84","Type":"ContainerStarted","Data":"d500785a407f9994f15a5ce867c6e9bf4a939b282aa7b14e5f1d9268ddf8a924"}
Nov 03 22:10:11.368842 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.368799    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" podStartSLOduration=1.368788436 podStartE2EDuration="1.368788436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:11.367900879 +0000 UTC m=+4912.999214356" watchObservedRunningTime="2026-11-03 22:10:11.368788436 +0000 UTC m=+4913.000101910"
Nov 03 22:10:11.478930 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:11.478890    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" podStartSLOduration=1.478878757 podStartE2EDuration="1.478878757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:11.476663646 +0000 UTC m=+4913.107977131" watchObservedRunningTime="2026-11-03 22:10:11.478878757 +0000 UTC m=+4913.110192234"
Nov 03 22:10:15.051734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:15.051697    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"]
Nov 03 22:10:15.100135 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:15.100096    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76ad99dc-85598"]
Nov 03 22:10:15.102181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:15.102158    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"]
Nov 03 22:10:15.104116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:15.104092    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c413af70-k5t6h"]
Nov 03 22:10:16.491352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:16.491318    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4808ad-6486-4660-bd87-961f0deda0cd" path="/var/lib/kubelet/pods/0e4808ad-6486-4660-bd87-961f0deda0cd/volumes"
Nov 03 22:10:16.491755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:16.491741    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9f51b6-7f22-453b-a8b3-62648e80ecc9" path="/var/lib/kubelet/pods/fe9f51b6-7f22-453b-a8b3-62648e80ecc9/volumes"
Nov 03 22:10:19.260039 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:10:19.260008    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:10:20.113174 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:20.113136    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:10:22.204189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:22.204155    5533 scope.go:122] "RemoveContainer" containerID="d44f6670763dfe21fe1f69dccfd3a3af0da90185119475957e36d9f683943d84"
Nov 03 22:10:22.213224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:22.213201    5533 scope.go:122] "RemoveContainer" containerID="9171985ac92ad9556e907eb781ee3dc683e37c9a494a383224c91d2fa5c0d972"
Nov 03 22:10:22.278110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:22.278080    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:22.278228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:22.278150    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:28.047220 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:28.047176    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"]
Nov 03 22:10:28.092417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:28.092386    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0732a06-wlct4"]
Nov 03 22:10:28.491858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:28.491822    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2" path="/var/lib/kubelet/pods/9b062cd7-d2c6-44ad-9199-d1b2c18d2cd2/volumes"
Nov 03 22:10:29.372378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.372302    5533 generic.go:356] "Generic (PLEG): container finished" podID="169d8aca-c814-4057-88f5-d0117bea9952" containerID="a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131" exitCode=0
Nov 03 22:10:29.372378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.372362    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerDied","Data":"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"}
Nov 03 22:10:29.372900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.372712    5533 scope.go:122] "RemoveContainer" containerID="a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"
Nov 03 22:10:29.373855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.373826    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:29.373946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.373931    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:29.373989 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:29.373970    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:10:30.382576 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:30.382543    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerStarted","Data":"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"}
Nov 03 22:10:30.471074 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:30.471025    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:30.471439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:30.471414    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 22:10:30.471606 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:30.471581    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 22:10:31.471628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:31.471589    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 22:10:31.472235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:31.472205    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 22:10:32.472659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:32.471965    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 22:10:32.472659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:32.472036    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 22:10:33.471995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:33.471955    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 22:10:33.472356 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:33.472324    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 22:10:34.470660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:34.467978    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 22:10:34.470660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:34.468038    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 22:10:35.224554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:35.224516    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:35.466213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:35.466180    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:35.467372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:35.467349    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:10:42.832685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.832653    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"]
Nov 03 22:10:42.836212 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.836193    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.889690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.889660    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.889878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.889800    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.890044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.889896    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4xq\" (UniqueName: \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.891329 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.891305    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"]
Nov 03 22:10:42.934767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.934742    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:10:42.934875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.934800    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:10:42.990876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.990845    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4xq\" (UniqueName: \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.991001 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.990906    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.991084 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.990997    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.991591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.991568    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:42.993306 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:42.993285    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:43.049261 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:43.049239    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4xq\" (UniqueName: \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\") pod \"maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:43.148303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:43.148250    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:43.511718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:43.511690    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"]
Nov 03 22:10:43.514534 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:10:43.514505    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4457cb6b_ef61_41ce_a3d6_2ad47a573a55.slice/crio-d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539 WatchSource:0}: Error finding container d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539: Status 404 returned error can't find the container with id d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539
Nov 03 22:10:44.458219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:44.458190    5533 generic.go:356] "Generic (PLEG): container finished" podID="4457cb6b-ef61-41ce-a3d6-2ad47a573a55" containerID="0901e1e63dc678c9e438ce4bab9fd3a57d470bd0e7f3c403a5f6294988345f22" exitCode=0
Nov 03 22:10:44.458538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:44.458285    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r" event={"ID":"4457cb6b-ef61-41ce-a3d6-2ad47a573a55","Type":"ContainerDied","Data":"0901e1e63dc678c9e438ce4bab9fd3a57d470bd0e7f3c403a5f6294988345f22"}
Nov 03 22:10:44.458538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:44.458343    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r" event={"ID":"4457cb6b-ef61-41ce-a3d6-2ad47a573a55","Type":"ContainerStarted","Data":"d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539"}
Nov 03 22:10:45.585616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.585595    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:45.713474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.713447    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\") pod \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") "
Nov 03 22:10:45.713622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.713511    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\") pod \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") "
Nov 03 22:10:45.713622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.713558    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\" (UniqueName: \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\") pod \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\" (UID: \"4457cb6b-ef61-41ce-a3d6-2ad47a573a55\") "
Nov 03 22:10:45.713855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.713831    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca" pod "4457cb6b-ef61-41ce-a3d6-2ad47a573a55" (UID: "4457cb6b-ef61-41ce-a3d6-2ad47a573a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:45.715541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.715488    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca" pod "4457cb6b-ef61-41ce-a3d6-2ad47a573a55" (UID: "4457cb6b-ef61-41ce-a3d6-2ad47a573a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:45.715623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.715578    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq" pod "4457cb6b-ef61-41ce-a3d6-2ad47a573a55" (UID: "4457cb6b-ef61-41ce-a3d6-2ad47a573a55"). InnerVolumeSpecName "kube-api-access-sv4xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:45.814258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.814235    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4xq\" (UniqueName: \"kubernetes.io/projected/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-kube-api-access-sv4xq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:45.814258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.814257    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:45.814384 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:45.814269    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4457cb6b-ef61-41ce-a3d6-2ad47a573a55-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:46.469810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:46.469781    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"
Nov 03 22:10:46.469977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:46.469773    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r" event={"ID":"4457cb6b-ef61-41ce-a3d6-2ad47a573a55","Type":"ContainerDied","Data":"d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539"}
Nov 03 22:10:46.469977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:46.469913    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6837718d0a43877c93c8180ec3c3eea5327b1655e7d5148e88a43fbf312d539"
Nov 03 22:10:49.259364 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:10:49.259330    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:10:54.331539 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:54.331504    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:10:54.442191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:54.442146    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:10:59.332652 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.332553    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" podUID="43b93452-e330-41b4-bd29-dcec97108bf9" containerName="payload-processing" containerID="cri-o://36b174b92bf828edb24ce7a79c1057745dc48e0e34f3e2e9018fcd5ef5624d19" gracePeriod=25
Nov 03 22:10:59.443231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.443158    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" podUID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" containerName="payload-pre-processing" containerID="cri-o://8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60" gracePeriod=25
Nov 03 22:10:59.542426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.542396    5533 generic.go:356] "Generic (PLEG): container finished" podID="43b93452-e330-41b4-bd29-dcec97108bf9" containerID="36b174b92bf828edb24ce7a79c1057745dc48e0e34f3e2e9018fcd5ef5624d19" exitCode=0
Nov 03 22:10:59.542572 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.542469    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" event={"ID":"43b93452-e330-41b4-bd29-dcec97108bf9","Type":"ContainerDied","Data":"36b174b92bf828edb24ce7a79c1057745dc48e0e34f3e2e9018fcd5ef5624d19"}
Nov 03 22:10:59.576905 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.576880    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:10:59.691559 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.691532    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:10:59.731076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.731043    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\") pod \"43b93452-e330-41b4-bd29-dcec97108bf9\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") "
Nov 03 22:10:59.731210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.731119    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\" (UniqueName: \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\") pod \"43b93452-e330-41b4-bd29-dcec97108bf9\" (UID: \"43b93452-e330-41b4-bd29-dcec97108bf9\") "
Nov 03 22:10:59.731401 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.731379    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume" pod "43b93452-e330-41b4-bd29-dcec97108bf9" (UID: "43b93452-e330-41b4-bd29-dcec97108bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:59.733183 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.733163    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4" pod "43b93452-e330-41b4-bd29-dcec97108bf9" (UID: "43b93452-e330-41b4-bd29-dcec97108bf9"). InnerVolumeSpecName "kube-api-access-lcvs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:59.832013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.831986    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\") pod \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") "
Nov 03 22:10:59.832147 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.832020    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\" (UniqueName: \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\") pod \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\" (UID: \"e1a383a7-dfb4-48d2-8d74-1b1a87903e84\") "
Nov 03 22:10:59.832250 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.832239    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvs4\" (UniqueName: \"kubernetes.io/projected/43b93452-e330-41b4-bd29-dcec97108bf9-kube-api-access-lcvs4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:59.832297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.832252    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b93452-e330-41b4-bd29-dcec97108bf9-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:59.832416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.832393    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume" pod "e1a383a7-dfb4-48d2-8d74-1b1a87903e84" (UID: "e1a383a7-dfb4-48d2-8d74-1b1a87903e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:59.833997 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.833977    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh" pod "e1a383a7-dfb4-48d2-8d74-1b1a87903e84" (UID: "e1a383a7-dfb4-48d2-8d74-1b1a87903e84"). InnerVolumeSpecName "kube-api-access-fjkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:59.869330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.869271    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"]
Nov 03 22:10:59.875900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.875880    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4edfb8c0-zcl7r"]
Nov 03 22:10:59.933593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.933567    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:10:59.933774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:10:59.933757    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkzh\" (UniqueName: \"kubernetes.io/projected/e1a383a7-dfb4-48d2-8d74-1b1a87903e84-kube-api-access-fjkzh\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:00.492032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.491996    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4457cb6b-ef61-41ce-a3d6-2ad47a573a55" path="/var/lib/kubelet/pods/4457cb6b-ef61-41ce-a3d6-2ad47a573a55/volumes"
Nov 03 22:11:00.549284 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.549246    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh" event={"ID":"43b93452-e330-41b4-bd29-dcec97108bf9","Type":"ContainerDied","Data":"b86e813f805c576d5347dc82729b2936b87c523d545fa9f0538cbbef995cd417"}
Nov 03 22:11:00.549284 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.549277    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"
Nov 03 22:11:00.549501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.549316    5533 scope.go:122] "RemoveContainer" containerID="36b174b92bf828edb24ce7a79c1057745dc48e0e34f3e2e9018fcd5ef5624d19"
Nov 03 22:11:00.550694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.550670    5533 generic.go:356] "Generic (PLEG): container finished" podID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" containerID="8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60" exitCode=0
Nov 03 22:11:00.550823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.550728    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"
Nov 03 22:11:00.550823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.550754    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" event={"ID":"e1a383a7-dfb4-48d2-8d74-1b1a87903e84","Type":"ContainerDied","Data":"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"}
Nov 03 22:11:00.550823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.550807    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw" event={"ID":"e1a383a7-dfb4-48d2-8d74-1b1a87903e84","Type":"ContainerDied","Data":"d500785a407f9994f15a5ce867c6e9bf4a939b282aa7b14e5f1d9268ddf8a924"}
Nov 03 22:11:00.559091 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.559060    5533 scope.go:122] "RemoveContainer" containerID="8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"
Nov 03 22:11:00.569959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.569937    5533 scope.go:122] "RemoveContainer" containerID="8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"
Nov 03 22:11:00.570245 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:00.570226    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60\": container with ID starting with 8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60 not found: ID does not exist" containerID="8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"
Nov 03 22:11:00.570325 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.570256    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60"} err="failed to get container status \"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60\": rpc error: code = NotFound desc = could not find container \"8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60\": container with ID starting with 8cd8d0e103c5aeaaac4f46f8d1d56831d4ed24684c1c19be44446290f728eb60 not found: ID does not exist"
Nov 03 22:11:00.703970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.703939    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:11:00.707406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.707382    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4edfb8c0-fb685c78-429lh"]
Nov 03 22:11:00.809714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.809687    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:11:00.813694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:00.813669    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4edfb8c0-8565b44c9f-t8bkw"]
Nov 03 22:11:02.491262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:02.491229    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b93452-e330-41b4-bd29-dcec97108bf9" path="/var/lib/kubelet/pods/43b93452-e330-41b4-bd29-dcec97108bf9/volumes"
Nov 03 22:11:02.491657 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:02.491601    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" path="/var/lib/kubelet/pods/e1a383a7-dfb4-48d2-8d74-1b1a87903e84/volumes"
Nov 03 22:11:02.491884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:02.491869    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:11:03.951522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:03.951483    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:11:03.951945 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:03.951703    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" containerID="cri-o://92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88" gracePeriod=30
Nov 03 22:11:05.468890 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:05.468852    5533 patch_prober.go:29] interesting pod/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:11:05.469382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:05.468927    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:11:07.052097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:07.052058    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"]
Nov 03 22:11:07.096780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:07.096746    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2a7c34-48mfs"]
Nov 03 22:11:08.493083 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:08.493045    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e" path="/var/lib/kubelet/pods/8f4082ee-7e09-4e86-b9a5-1660d1ca6b6e/volumes"
Nov 03 22:11:09.197273 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.197248    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:11:09.211926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.211898    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.211983    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212049    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212106    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212123    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:09.212317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212149    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212201    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.212428 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212351    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:11:09.212488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212474    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:09.212561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212487    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/169d8aca-c814-4057-88f5-d0117bea9952-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.212561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.212507    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.214096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.214076    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:09.214222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.214175    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "kube-api-access-b2cc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:09.214299 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.214266    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:11:09.214348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.214287    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:09.313470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313450    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.313608 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313477    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\") pod \"169d8aca-c814-4057-88f5-d0117bea9952\" (UID: \"169d8aca-c814-4057-88f5-d0117bea9952\") "
Nov 03 22:11:09.313716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313694    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.313716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313710    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/169d8aca-c814-4057-88f5-d0117bea9952-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.313815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313719    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.313815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313728    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2cc6\" (UniqueName: \"kubernetes.io/projected/169d8aca-c814-4057-88f5-d0117bea9952-kube-api-access-b2cc6\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.313815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313738    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.313815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313734    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:09.313815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.313769    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket" pod "169d8aca-c814-4057-88f5-d0117bea9952" (UID: "169d8aca-c814-4057-88f5-d0117bea9952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:09.414259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.414229    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.414259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.414254    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/169d8aca-c814-4057-88f5-d0117bea9952-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:09.602313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.602245    5533 generic.go:356] "Generic (PLEG): container finished" podID="169d8aca-c814-4057-88f5-d0117bea9952" containerID="92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88" exitCode=0
Nov 03 22:11:09.602313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.602303    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"
Nov 03 22:11:09.602865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.602318    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerDied","Data":"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"}
Nov 03 22:11:09.602865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.602372    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7" event={"ID":"169d8aca-c814-4057-88f5-d0117bea9952","Type":"ContainerDied","Data":"01c8902e94e91b1771358782dbe18bc7b53973dec7614f477f0465fc10c5b77b"}
Nov 03 22:11:09.602865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.602395    5533 scope.go:122] "RemoveContainer" containerID="92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"
Nov 03 22:11:09.612132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.612113    5533 scope.go:122] "RemoveContainer" containerID="a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"
Nov 03 22:11:09.620034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.620015    5533 scope.go:122] "RemoveContainer" containerID="92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"
Nov 03 22:11:09.620310 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:09.620289    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88\": container with ID starting with 92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88 not found: ID does not exist" containerID="92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"
Nov 03 22:11:09.620360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.620319    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88"} err="failed to get container status \"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88\": rpc error: code = NotFound desc = could not find container \"92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88\": container with ID starting with 92980b820a1c1aa56eddd88e5de2cffae29b0f24991063510d883360ee48bd88 not found: ID does not exist"
Nov 03 22:11:09.620360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.620346    5533 scope.go:122] "RemoveContainer" containerID="a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"
Nov 03 22:11:09.620606 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:09.620590    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131\": container with ID starting with a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131 not found: ID does not exist" containerID="a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"
Nov 03 22:11:09.620681 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.620613    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131"} err="failed to get container status \"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131\": rpc error: code = NotFound desc = could not find container \"a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131\": container with ID starting with a526fb3f85f7b2a0c77fff1f20291a3a45ce3227655ac84573ee6b1643d27131 not found: ID does not exist"
Nov 03 22:11:09.758106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.758077    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:11:09.805148 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:09.805125    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4edfb8c0-openshift-default-75647d8577-wmnv7"]
Nov 03 22:11:10.251625 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.251591    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:10.251962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.251948    5533 cpu_manager.go:414] "containerMap: removing container" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.251964    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.251989    5533 cpu_manager.go:414] "containerMap: removing container" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.251994    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252005    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4457cb6b-ef61-41ce-a3d6-2ad47a573a55" containerName="revoke-keys"
Nov 03 22:11:10.252021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252010    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4457cb6b-ef61-41ce-a3d6-2ad47a573a55" containerName="revoke-keys"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252029    5533 cpu_manager.go:414] "containerMap: removing container" podUID="43b93452-e330-41b4-bd29-dcec97108bf9" containerName="payload-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252034    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b93452-e330-41b4-bd29-dcec97108bf9" containerName="payload-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252041    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" containerName="payload-pre-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252046    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" containerName="payload-pre-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252097    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b93452-e330-41b4-bd29-dcec97108bf9" containerName="payload-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252105    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a383a7-dfb4-48d2-8d74-1b1a87903e84" containerName="payload-pre-processing"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252113    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252120    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="169d8aca-c814-4057-88f5-d0117bea9952" containerName="istio-proxy"
Nov 03 22:11:10.252191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.252126    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4457cb6b-ef61-41ce-a3d6-2ad47a573a55" containerName="revoke-keys"
Nov 03 22:11:10.256609 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.256588    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.306095 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306076    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 22:11:10.306209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306189    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 22:11:10.306209 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306201    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 22:11:10.306297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306242    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 22:11:10.306297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306255    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9vrxc\""
Nov 03 22:11:10.306297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.306257    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 22:11:10.312460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.312440    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:10.322054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.322033    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kqb\" (UniqueName: \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.322159 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.322096    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.322213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.322197    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.423185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.423164    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.423283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.423202    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.423283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.423223    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kqb\" (UniqueName: \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.425532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.425508    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.425688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.425668    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.480629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.480606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kqb\" (UniqueName: \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\") pod \"maas-controller-77b845648-8skw8\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") " pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.491709 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.491688    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169d8aca-c814-4057-88f5-d0117bea9952" path="/var/lib/kubelet/pods/169d8aca-c814-4057-88f5-d0117bea9952/volumes"
Nov 03 22:11:10.569213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.569169    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:10.936436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.936413    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:10.937698 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:11:10.937673    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e5a2a7_da44_464c_9ed9_fba7b44bd1cf.slice/crio-16fefc8ec0e09191be982fc24ef89adad0309c9ef30832862a18f0e80f90e626 WatchSource:0}: Error finding container 16fefc8ec0e09191be982fc24ef89adad0309c9ef30832862a18f0e80f90e626: Status 404 returned error can't find the container with id 16fefc8ec0e09191be982fc24ef89adad0309c9ef30832862a18f0e80f90e626
Nov 03 22:11:10.939119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:10.939101    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:11:11.617008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:11.616973    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-8skw8" event={"ID":"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf","Type":"ContainerStarted","Data":"16fefc8ec0e09191be982fc24ef89adad0309c9ef30832862a18f0e80f90e626"}
Nov 03 22:11:16.648010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:16.647971    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-8skw8" event={"ID":"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf","Type":"ContainerStarted","Data":"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"}
Nov 03 22:11:16.648502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:16.648154    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:16.762374 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:16.762320    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-77b845648-8skw8" podStartSLOduration=1.820254381 podStartE2EDuration="6.762307265s" totalImagesPullingTime="4.942052884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:16.759823148 +0000 UTC m=+4978.391136625" watchObservedRunningTime="2026-11-03 22:11:16.762307265 +0000 UTC m=+4978.393620733"
Nov 03 22:11:18.046153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:18.046119    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"]
Nov 03 22:11:18.092484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:18.092456    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ba485-95ssm"]
Nov 03 22:11:18.492516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:18.492468    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f61bb42-4c0a-4d95-92f7-687b8d2d6320" path="/var/lib/kubelet/pods/6f61bb42-4c0a-4d95-92f7-687b8d2d6320/volumes"
Nov 03 22:11:19.259505 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:19.259471    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:11:22.249791 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:22.249756    5533 scope.go:122] "RemoveContainer" containerID="0bf2fcd36bad1ba071c11a507740e5b7df9ce352f8495b7c796c3c72564ceb4c"
Nov 03 22:11:22.258319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:22.258297    5533 scope.go:122] "RemoveContainer" containerID="b585a74243e46f5fae89d6108ca03b9e9e58633562fe2ef5653bc9e16a9db42f"
Nov 03 22:11:22.266242 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:22.266224    5533 scope.go:122] "RemoveContainer" containerID="3a41f2564d6ad256e975f3dd62831f17c3f5e51df599cdd89b13b4c04247e9c8"
Nov 03 22:11:27.657241 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.657207    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:27.904769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.904736    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"]
Nov 03 22:11:27.910307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.910239    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.959937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.959915    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4cjrg\""
Nov 03 22:11:27.962165 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962140    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnds\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-kube-api-access-7vnds\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962192    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962283    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962328    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962392 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962350    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962401    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d70fc5c-1dba-4651-b45b-299fa16b1692-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962441    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962484    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.962519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.962511    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:27.967093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:27.967072    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"]
Nov 03 22:11:28.063617 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063590    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnds\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-kube-api-access-7vnds\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063628    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063681    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063707    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063725    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063742    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d70fc5c-1dba-4651-b45b-299fa16b1692-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.063777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063766    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064134 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063791    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064134 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.063817    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.064141    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.064189    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064364 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.064277    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064364 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.064336    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.064553 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.064535    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d70fc5c-1dba-4651-b45b-299fa16b1692-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.066017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.065999    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.066265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.066245    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.121389 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.121362    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.121701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.121681    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnds\" (UniqueName: \"kubernetes.io/projected/6d70fc5c-1dba-4651-b45b-299fa16b1692-kube-api-access-7vnds\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-7lrp6\" (UID: \"6d70fc5c-1dba-4651-b45b-299fa16b1692\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.223416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.223393    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:28.639381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.639309    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"]
Nov 03 22:11:28.674346 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:11:28.674322    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d70fc5c_1dba_4651_b45b_299fa16b1692.slice/crio-0be00cfc1f1519961b1d54a2b0c34c6d6674f6bde5ba1a25a945bcb19f22116a WatchSource:0}: Error finding container 0be00cfc1f1519961b1d54a2b0c34c6d6674f6bde5ba1a25a945bcb19f22116a: Status 404 returned error can't find the container with id 0be00cfc1f1519961b1d54a2b0c34c6d6674f6bde5ba1a25a945bcb19f22116a
Nov 03 22:11:28.676334 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.676301    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:28.676429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.676362    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:28.676429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.676393    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:28.713076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:28.713050    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" event={"ID":"6d70fc5c-1dba-4651-b45b-299fa16b1692","Type":"ContainerStarted","Data":"0be00cfc1f1519961b1d54a2b0c34c6d6674f6bde5ba1a25a945bcb19f22116a"}
Nov 03 22:11:29.719810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:29.719771    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" event={"ID":"6d70fc5c-1dba-4651-b45b-299fa16b1692","Type":"ContainerStarted","Data":"428f4b3917336aea21e2b0f8ae261fe5fb9a92ce0b40afc96aba28d078181ec4"}
Nov 03 22:11:29.888824 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:29.888768    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podStartSLOduration=2.888753591 podStartE2EDuration="2.888753591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:29.886759872 +0000 UTC m=+4991.518073350" watchObservedRunningTime="2026-11-03 22:11:29.888753591 +0000 UTC m=+4991.520067068"
Nov 03 22:11:30.224236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:30.224209    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:30.225680 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:30.225656    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:30.225816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:30.225699    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:31.224459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:31.224425    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:31.224960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:31.224481    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:32.224326 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:32.224288    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:32.224515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:32.224340    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:33.224551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:33.224517    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:33.224987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:33.224571    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:34.224243 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.224208    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:34.224507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.224265    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:34.569799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.569722    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"]
Nov 03 22:11:34.573837 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.573816    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.623368 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.623326    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.623542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.623387    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2trt\" (UniqueName: \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.623586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.623551    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.624400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.624366    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:11:34.670829 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.670799    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:11:34.672963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.672942    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"]
Nov 03 22:11:34.724965 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.724942    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2trt\" (UniqueName: \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.725087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.725003    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.725087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.725049    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.725667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.725626    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.727471 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.727446    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.782717 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.782682    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2trt\" (UniqueName: \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\") pod \"maas-api-revoke-sub-85237e45-rwsf8\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:34.887527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:34.887456    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:35.224239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.224202    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused" start-of-body=
Nov 03 22:11:35.224458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.224269    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": dial tcp 10.133.1.94:15021: connect: connection refused"
Nov 03 22:11:35.254886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.254856    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"]
Nov 03 22:11:35.257875 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:11:35.257844    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e76e4d6_e2cd_416a_9146_b3d09dbf5c37.slice/crio-6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a WatchSource:0}: Error finding container 6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a: Status 404 returned error can't find the container with id 6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a
Nov 03 22:11:35.754706 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.754661    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8" event={"ID":"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37","Type":"ContainerStarted","Data":"97b78c2f32e4dd08e00561bc433692654dbec521d162eb0607dffe750cb912c4"}
Nov 03 22:11:35.755074 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.754737    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8" event={"ID":"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37","Type":"ContainerStarted","Data":"6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a"}
Nov 03 22:11:35.866179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:35.866133    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8" podStartSLOduration=1.8661179140000002 podStartE2EDuration="1.866117914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:35.863950829 +0000 UTC m=+4997.495264305" watchObservedRunningTime="2026-11-03 22:11:35.866117914 +0000 UTC m=+4997.497431431"
Nov 03 22:11:36.762466 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:36.762437    5533 generic.go:356] "Generic (PLEG): container finished" podID="0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" containerID="97b78c2f32e4dd08e00561bc433692654dbec521d162eb0607dffe750cb912c4" exitCode=0
Nov 03 22:11:36.762947 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:36.762524    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8" event={"ID":"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37","Type":"ContainerDied","Data":"97b78c2f32e4dd08e00561bc433692654dbec521d162eb0607dffe750cb912c4"}
Nov 03 22:11:37.056282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.056143    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:37.056786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.056754    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:37.057868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.057848    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6"
Nov 03 22:11:37.890529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.890506    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:37.954000 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.953971    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\") pod \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") "
Nov 03 22:11:37.954128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.954035    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\") pod \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") "
Nov 03 22:11:37.954172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.954154    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\" (UniqueName: \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\") pod \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\" (UID: \"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37\") "
Nov 03 22:11:37.954393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.954367    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca" pod "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" (UID: "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:11:37.954469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.954444    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:37.955955 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.955936    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca" pod "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" (UID: "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:11:37.956007 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:37.955985    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt" pod "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" (UID: "0e76e4d6-e2cd-416a-9146-b3d09dbf5c37"). InnerVolumeSpecName "kube-api-access-h2trt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:38.055433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:38.055371    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:38.055433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:38.055396    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2trt\" (UniqueName: \"kubernetes.io/projected/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37-kube-api-access-h2trt\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:38.773892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:38.773813    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"
Nov 03 22:11:38.773892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:38.773850    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8" event={"ID":"0e76e4d6-e2cd-416a-9146-b3d09dbf5c37","Type":"ContainerDied","Data":"6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a"}
Nov 03 22:11:38.774091 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:38.773897    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ceb630efb6e26fc141a2657a871284c81aaeb2197103fdbc3f810b207a0801a"
Nov 03 22:11:43.094586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.094551    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"]
Nov 03 22:11:43.094994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.094905    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" containerName="revoke-keys"
Nov 03 22:11:43.094994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.094917    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" containerName="revoke-keys"
Nov 03 22:11:43.095085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.094998    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" containerName="revoke-keys"
Nov 03 22:11:43.101094 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.101069    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.169014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.168990    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"]
Nov 03 22:11:43.198606 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198572    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.198606 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198606    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.198831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198627    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.198831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198728    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.198831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198780    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.198831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198829    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcdl\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-kube-api-access-fgcdl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.199052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198853    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25810e8f-c406-4e5b-9194-64d849a8c95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.199052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198925    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.199052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.198945    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299456    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcdl\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-kube-api-access-fgcdl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299487    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25810e8f-c406-4e5b-9194-64d849a8c95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299623 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299509    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299665    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299746    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299806 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299775    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299986 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299807    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299986 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299837    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.299986 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.299876    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.300152 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.300097    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.300282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.300259    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.300333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.300290    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.300383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.300358    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25810e8f-c406-4e5b-9194-64d849a8c95e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.300567 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.300546    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.301883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.301863    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.302236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.302214    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.357670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.357594    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.357765 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.357726    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcdl\" (UniqueName: \"kubernetes.io/projected/25810e8f-c406-4e5b-9194-64d849a8c95e-kube-api-access-fgcdl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-5swkz\" (UID: \"25810e8f-c406-4e5b-9194-64d849a8c95e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.415179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.415157    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:43.830369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.830344    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"]
Nov 03 22:11:43.832309 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:11:43.832280    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25810e8f_c406_4e5b_9194_64d849a8c95e.slice/crio-e9cfad02176a64a484c50087ea6e9ba5bf421dff873a917170a98ee58da5f07c WatchSource:0}: Error finding container e9cfad02176a64a484c50087ea6e9ba5bf421dff873a917170a98ee58da5f07c: Status 404 returned error can't find the container with id e9cfad02176a64a484c50087ea6e9ba5bf421dff873a917170a98ee58da5f07c
Nov 03 22:11:43.834244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.834212    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:43.834354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.834281    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:43.834354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:43.834310    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:11:44.809894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:44.809856    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" event={"ID":"25810e8f-c406-4e5b-9194-64d849a8c95e","Type":"ContainerStarted","Data":"672353336e27dfb9e7d9ea809bc3f9dd8b77f2183d0740ea4957cf28b9dbdf38"}
Nov 03 22:11:44.810308 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:44.809917    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" event={"ID":"25810e8f-c406-4e5b-9194-64d849a8c95e","Type":"ContainerStarted","Data":"e9cfad02176a64a484c50087ea6e9ba5bf421dff873a917170a98ee58da5f07c"}
Nov 03 22:11:45.027297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:45.027240    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podStartSLOduration=2.027226578 podStartE2EDuration="2.027226578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:44.980263226 +0000 UTC m=+5006.611576732" watchObservedRunningTime="2026-11-03 22:11:45.027226578 +0000 UTC m=+5006.658540055"
Nov 03 22:11:45.416124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:45.416091    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:45.417676 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:45.417650    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:45.417835 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:45.417716    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:46.416127 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:46.416091    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:46.416549 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:46.416145    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:47.288277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.288242    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:47.288462 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.288435    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-77b845648-8skw8" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerName="manager" containerID="cri-o://074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2" gracePeriod=30
Nov 03 22:11:47.416259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.416232    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:47.416596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.416279    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:47.681658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.680629    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:47.739251 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.739216    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\" (UniqueName: \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\") pod \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") "
Nov 03 22:11:47.739423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.739280    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\") pod \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") "
Nov 03 22:11:47.739423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.739340    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\") pod \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\" (UID: \"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf\") "
Nov 03 22:11:47.741334 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.741300    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb" pod "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" (UID: "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf"). InnerVolumeSpecName "kube-api-access-z9kqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:47.741492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.741394    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs" pod "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" (UID: "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:11:47.741554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.741486    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert" pod "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" (UID: "b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:11:47.829666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.829565    5533 generic.go:356] "Generic (PLEG): container finished" podID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerID="074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2" exitCode=0
Nov 03 22:11:47.829666 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.829609    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-8skw8"
Nov 03 22:11:47.829883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.829693    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-8skw8" event={"ID":"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf","Type":"ContainerDied","Data":"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"}
Nov 03 22:11:47.829883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.829750    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-8skw8" event={"ID":"b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf","Type":"ContainerDied","Data":"16fefc8ec0e09191be982fc24ef89adad0309c9ef30832862a18f0e80f90e626"}
Nov 03 22:11:47.829883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.829777    5533 scope.go:122] "RemoveContainer" containerID="074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"
Nov 03 22:11:47.839908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.839885    5533 scope.go:122] "RemoveContainer" containerID="074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"
Nov 03 22:11:47.840198 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.840173    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kqb\" (UniqueName: \"kubernetes.io/projected/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-kube-api-access-z9kqb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.840329 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.840204    5533 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-webhook-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.840329 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:47.840204    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2\": container with ID starting with 074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2 not found: ID does not exist" containerID="074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"
Nov 03 22:11:47.840329 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.840219    5533 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf-metrics-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:11:47.840329 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.840233    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2"} err="failed to get container status \"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2\": rpc error: code = NotFound desc = could not find container \"074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2\": container with ID starting with 074a896914048e1dced531afb895fe61b2770252ae0909c7160514e0a36039b2 not found: ID does not exist"
Nov 03 22:11:47.985399 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.985366    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:47.992437 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:47.990941    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77b845648-8skw8"]
Nov 03 22:11:48.045217 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.045190    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"]
Nov 03 22:11:48.048171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.048146    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcd1f14-hbczl"]
Nov 03 22:11:48.415854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.415817    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:48.416039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.415881    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:48.491433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.491395    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" path="/var/lib/kubelet/pods/b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf/volumes"
Nov 03 22:11:48.492005 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.491985    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b0cd80-993f-44a5-b89d-fff411cf82b7" path="/var/lib/kubelet/pods/d2b0cd80-993f-44a5-b89d-fff411cf82b7/volumes"
Nov 03 22:11:48.665744 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:48.665708    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-77b845648-8skw8" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerName="manager" probeResult="failure" output="Get \"http://10.133.1.93:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:11:49.285424 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:11:49.285384    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:11:49.416468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.416435    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:49.416699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.416491    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:49.474655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.474486    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77b845648-rzlw2"]
Nov 03 22:11:49.475658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.475089    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerName="manager"
Nov 03 22:11:49.475658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.475125    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerName="manager"
Nov 03 22:11:49.475658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.475312    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e5a2a7-da44-464c-9ed9-fba7b44bd1cf" containerName="manager"
Nov 03 22:11:49.480757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.480737    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.530096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.530075    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 22:11:49.576496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.575843    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9vrxc\""
Nov 03 22:11:49.576496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.576362    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 22:11:49.576496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.576410    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 22:11:49.577674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.577237    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 22:11:49.577674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.577359    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 22:11:49.580782 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.580757    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-rzlw2"]
Nov 03 22:11:49.581150 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.581122    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnf8j\" (UniqueName: \"kubernetes.io/projected/7e9c8826-0e09-4d08-836c-186eab92e73e-kube-api-access-rnf8j\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.581277 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.581252    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-webhook-cert\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.581346 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.581322    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-metrics-certs\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.682468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.682426    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-webhook-cert\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.682468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.682472    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-metrics-certs\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.682812 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.682522    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnf8j\" (UniqueName: \"kubernetes.io/projected/7e9c8826-0e09-4d08-836c-186eab92e73e-kube-api-access-rnf8j\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.685151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.685119    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-webhook-cert\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.685301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.685258    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9c8826-0e09-4d08-836c-186eab92e73e-metrics-certs\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.741288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.741252    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnf8j\" (UniqueName: \"kubernetes.io/projected/7e9c8826-0e09-4d08-836c-186eab92e73e-kube-api-access-rnf8j\") pod \"maas-controller-77b845648-rzlw2\" (UID: \"7e9c8826-0e09-4d08-836c-186eab92e73e\") " pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:49.793681 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:49.793657    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:50.163308 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:50.163168    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77b845648-rzlw2"]
Nov 03 22:11:50.166077 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:11:50.166046    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9c8826_0e09_4d08_836c_186eab92e73e.slice/crio-f6250e75488c8f15e02ccc05285535fa98844940751dd1e1862f6e5d8c42dd90 WatchSource:0}: Error finding container f6250e75488c8f15e02ccc05285535fa98844940751dd1e1862f6e5d8c42dd90: Status 404 returned error can't find the container with id f6250e75488c8f15e02ccc05285535fa98844940751dd1e1862f6e5d8c42dd90
Nov 03 22:11:50.416012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:50.415952    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused" start-of-body=
Nov 03 22:11:50.416012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:50.416001    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": dial tcp 10.133.1.96:15021: connect: connection refused"
Nov 03 22:11:50.851971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:50.851940    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-rzlw2" event={"ID":"7e9c8826-0e09-4d08-836c-186eab92e73e","Type":"ContainerStarted","Data":"f6250e75488c8f15e02ccc05285535fa98844940751dd1e1862f6e5d8c42dd90"}
Nov 03 22:11:51.861089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:51.861053    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-rzlw2" event={"ID":"7e9c8826-0e09-4d08-836c-186eab92e73e","Type":"ContainerStarted","Data":"633055ff35614bde497982c504536cef3ab5ca458f352960ed63011aa1e81d6b"}
Nov 03 22:11:51.861525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:51.861189    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:11:51.974421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:51.974376    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-77b845648-rzlw2" podStartSLOduration=2.077878981 podStartE2EDuration="2.974362528s" totalImagesPullingTime="896.483547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:51.972586859 +0000 UTC m=+5013.603900336" watchObservedRunningTime="2026-11-03 22:11:51.974362528 +0000 UTC m=+5013.605676031"
Nov 03 22:11:52.100803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:52.100764    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:52.101165 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:52.101142    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:52.102087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:52.102066    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz"
Nov 03 22:11:59.045269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:59.045234    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"]
Nov 03 22:11:59.091921 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:11:59.091894    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876f9649-cnfvf"]
Nov 03 22:12:00.491772 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:00.491739    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89711c04-b751-4817-8267-391f384c843d" path="/var/lib/kubelet/pods/89711c04-b751-4817-8267-391f384c843d/volumes"
Nov 03 22:12:02.868531 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:02.868498    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:12:10.048248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:10.048211    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"]
Nov 03 22:12:10.094687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:10.094658    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a155d2-x9zmw"]
Nov 03 22:12:10.492333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:10.492303    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9fb82f-d236-42e2-8615-14c7f0e05e96" path="/var/lib/kubelet/pods/9a9fb82f-d236-42e2-8615-14c7f0e05e96/volumes"
Nov 03 22:12:15.517685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:15.517645    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:12:19.259545 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:12:19.259510    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:12:21.055357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.055324    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"]
Nov 03 22:12:21.102605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.102577    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181c96ac-tcxfj"]
Nov 03 22:12:21.721421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.721386    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"]
Nov 03 22:12:21.726185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.726165    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.747375 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.747347    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.747492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.747389    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.747535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.747500    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mhs\" (UniqueName: \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.779086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.779053    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:12:21.779086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.779072    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:12:21.779648 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.779621    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"]
Nov 03 22:12:21.848894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.848873    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.849003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.848913    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.849003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.848957    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mhs\" (UniqueName: \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.849488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.849466    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.851275 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.851256    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:21.906579 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:21.906556    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mhs\" (UniqueName: \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\") pod \"maas-api-revoke-sub-f904bff9-zxf69\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:22.038524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.038470    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:22.306586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.306515    5533 scope.go:122] "RemoveContainer" containerID="bd0bf182bb828d917c8c28e2cee82c503a3e9fd41b0977a036eb92fede65a324"
Nov 03 22:12:22.315186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.315164    5533 scope.go:122] "RemoveContainer" containerID="6c1d0a9853ba003f31edc62f7299c4d3f6cbee85d2e900143042a521aa510466"
Nov 03 22:12:22.322859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.322842    5533 scope.go:122] "RemoveContainer" containerID="d980eb92ada9fc0917046bb66f2ba85d6b72c3185390c73aea716b4a54347fdf"
Nov 03 22:12:22.402076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.402054    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"]
Nov 03 22:12:22.406124 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:12:22.406090    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c50f75b_8632_406b_8796_d4776752cf53.slice/crio-05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f WatchSource:0}: Error finding container 05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f: Status 404 returned error can't find the container with id 05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f
Nov 03 22:12:22.493038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:22.493011    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d8ac77-64f9-47eb-95db-f929a67b7e0c" path="/var/lib/kubelet/pods/26d8ac77-64f9-47eb-95db-f929a67b7e0c/volumes"
Nov 03 22:12:23.034530 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:23.034493    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69" event={"ID":"6c50f75b-8632-406b-8796-d4776752cf53","Type":"ContainerStarted","Data":"8eb93d50a448173d97381cf764282ca1d32e0584f4d4064818731a7b46d29509"}
Nov 03 22:12:23.034723 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:23.034546    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69" event={"ID":"6c50f75b-8632-406b-8796-d4776752cf53","Type":"ContainerStarted","Data":"05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f"}
Nov 03 22:12:23.145213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:23.145162    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69" podStartSLOduration=2.145148475 podStartE2EDuration="2.145148475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:23.14301293 +0000 UTC m=+5044.774326408" watchObservedRunningTime="2026-11-03 22:12:23.145148475 +0000 UTC m=+5044.776461942"
Nov 03 22:12:24.041507 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:24.041468    5533 generic.go:356] "Generic (PLEG): container finished" podID="6c50f75b-8632-406b-8796-d4776752cf53" containerID="8eb93d50a448173d97381cf764282ca1d32e0584f4d4064818731a7b46d29509" exitCode=0
Nov 03 22:12:24.041902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:24.041554    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69" event={"ID":"6c50f75b-8632-406b-8796-d4776752cf53","Type":"ContainerDied","Data":"8eb93d50a448173d97381cf764282ca1d32e0584f4d4064818731a7b46d29509"}
Nov 03 22:12:25.175168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.175146    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:25.279512 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.279488    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\" (UniqueName: \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\") pod \"6c50f75b-8632-406b-8796-d4776752cf53\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") "
Nov 03 22:12:25.279627 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.279521    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\") pod \"6c50f75b-8632-406b-8796-d4776752cf53\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") "
Nov 03 22:12:25.279627 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.279571    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\") pod \"6c50f75b-8632-406b-8796-d4776752cf53\" (UID: \"6c50f75b-8632-406b-8796-d4776752cf53\") "
Nov 03 22:12:25.279948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.279916    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca" pod "6c50f75b-8632-406b-8796-d4776752cf53" (UID: "6c50f75b-8632-406b-8796-d4776752cf53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:12:25.281540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.281518    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca" pod "6c50f75b-8632-406b-8796-d4776752cf53" (UID: "6c50f75b-8632-406b-8796-d4776752cf53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:12:25.281610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.281541    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs" pod "6c50f75b-8632-406b-8796-d4776752cf53" (UID: "6c50f75b-8632-406b-8796-d4776752cf53"). InnerVolumeSpecName "kube-api-access-b4mhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:12:25.380522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.380474    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mhs\" (UniqueName: \"kubernetes.io/projected/6c50f75b-8632-406b-8796-d4776752cf53-kube-api-access-b4mhs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:25.380522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.380496    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:25.380522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:25.380506    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c50f75b-8632-406b-8796-d4776752cf53-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:26.053613 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:26.053583    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"
Nov 03 22:12:26.053797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:26.053616    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69" event={"ID":"6c50f75b-8632-406b-8796-d4776752cf53","Type":"ContainerDied","Data":"05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f"}
Nov 03 22:12:26.053797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:26.053707    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c806508bc8d57c6865cd627c6066442011244f8f356e98385146969584d39f"
Nov 03 22:12:28.409463 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.409430    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"]
Nov 03 22:12:28.409894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.409801    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6c50f75b-8632-406b-8796-d4776752cf53" containerName="revoke-keys"
Nov 03 22:12:28.409894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.409814    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c50f75b-8632-406b-8796-d4776752cf53" containerName="revoke-keys"
Nov 03 22:12:28.409894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.409891    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c50f75b-8632-406b-8796-d4776752cf53" containerName="revoke-keys"
Nov 03 22:12:28.413259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.413240    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504146 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504120    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504157    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504190    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504261    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504370    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmw8\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-kube-api-access-lpmw8\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504410    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504530    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.504590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.504563    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605098    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmw8\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-kube-api-access-lpmw8\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605174    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605221    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605291    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605344    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.605873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605777    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606023 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.605999    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606058    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606132    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606167    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606306 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606281    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606464    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.606630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.606609    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.607672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.607618    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.608379 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.608356    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"]
Nov 03 22:12:28.608751 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.608727    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.609085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.609061    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.662533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.662482    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.662769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.662749    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmw8\" (UniqueName: \"kubernetes.io/projected/fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67-kube-api-access-lpmw8\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-z2qhq\" (UID: \"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:28.726783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:28.726761    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:29.044977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.044945    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"]
Nov 03 22:12:29.050878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.050848    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d972da-fxfb2"]
Nov 03 22:12:29.141833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.141748    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"]
Nov 03 22:12:29.144062 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:12:29.144038    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea9c0e1_cbc2_4b28_a4e0_ca51f3ed6a67.slice/crio-7df70b926033448a933626ed9a0497f99de6b9843e554094ae11313a10971184 WatchSource:0}: Error finding container 7df70b926033448a933626ed9a0497f99de6b9843e554094ae11313a10971184: Status 404 returned error can't find the container with id 7df70b926033448a933626ed9a0497f99de6b9843e554094ae11313a10971184
Nov 03 22:12:29.146138 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.146103    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:12:29.146233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.146164    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:12:29.146233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:29.146195    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:12:30.085009 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.084971    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" event={"ID":"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67","Type":"ContainerStarted","Data":"ba002c4658323d7c4a77ac023833697d886e3f19b12e5a77f0605e2a967d0882"}
Nov 03 22:12:30.085551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.085053    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" event={"ID":"fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67","Type":"ContainerStarted","Data":"7df70b926033448a933626ed9a0497f99de6b9843e554094ae11313a10971184"}
Nov 03 22:12:30.246980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.246922    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podStartSLOduration=2.246905072 podStartE2EDuration="2.246905072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:30.246379139 +0000 UTC m=+5051.877692618" watchObservedRunningTime="2026-11-03 22:12:30.246905072 +0000 UTC m=+5051.878218551"
Nov 03 22:12:30.491388 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.491350    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4e5958-823c-4a13-a8eb-f39de41e3e43" path="/var/lib/kubelet/pods/1e4e5958-823c-4a13-a8eb-f39de41e3e43/volumes"
Nov 03 22:12:30.727598 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.727561    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:30.728887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.728864    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:30.728952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:30.728912    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:31.727419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:31.727387    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:31.727830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:31.727444    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:32.727777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:32.727742    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:32.728160 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:32.727793    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:33.727360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:33.727325    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:33.727537 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:33.727376    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:34.727743 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:34.727695    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:34.728145 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:34.727769    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:34.803059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:34.803022    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:12:35.727789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:35.727752    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body=
Nov 03 22:12:35.728162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:35.727818    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused"
Nov 03 22:12:37.727751 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:37.727701    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:12:37.728255 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:37.727830    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:12:38.768788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:38.768746    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:12:38.769320 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:38.768793    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:12:39.627997 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:39.627965    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:39.628287 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:39.628268    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:39.628894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:39.628876    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq"
Nov 03 22:12:41.047819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:41.047783    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"]
Nov 03 22:12:41.094070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:41.094033    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb9ef22-rrb6x"]
Nov 03 22:12:42.520659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:42.515831    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea81498-b4ba-468e-927d-60f41941414a" path="/var/lib/kubelet/pods/0ea81498-b4ba-468e-927d-60f41941414a/volumes"
Nov 03 22:12:43.117291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:43.117255    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:12:49.265895 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:12:49.265829    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:12:53.397980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:53.397945    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:12:55.324933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.324901    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"]
Nov 03 22:12:55.329510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.329491    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.362073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.362046    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.362176 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.362088    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.362176 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.362161    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5jl\" (UniqueName: \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.429761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.429732    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:12:55.429917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.429902    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:12:55.431960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.431939    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"]
Nov 03 22:12:55.463589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.463563    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.463707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.463613    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.463707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.463668    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5jl\" (UniqueName: \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.464275 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.464254    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.466044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.466022    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.521461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.521429    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5jl\" (UniqueName: \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\") pod \"maas-api-revoke-sub-f03629db-t78fh\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:55.642184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:55.642122    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:56.008310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:56.008273    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"]
Nov 03 22:12:56.008741 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:12:56.008716    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7273cdc7_2f31_403d_b4c8_3027674f87b4.slice/crio-f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032 WatchSource:0}: Error finding container f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032: Status 404 returned error can't find the container with id f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032
Nov 03 22:12:56.351505 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:56.351473    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh" event={"ID":"7273cdc7-2f31-403d-b4c8-3027674f87b4","Type":"ContainerStarted","Data":"b3473f77e8678869a736239e832ead8a6574323e3a68b59ffe9fa80606eb625e"}
Nov 03 22:12:56.351883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:56.351539    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh" event={"ID":"7273cdc7-2f31-403d-b4c8-3027674f87b4","Type":"ContainerStarted","Data":"f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032"}
Nov 03 22:12:56.462195 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:56.462151    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh" podStartSLOduration=1.46213745 podStartE2EDuration="1.46213745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:56.460157438 +0000 UTC m=+5078.091470919" watchObservedRunningTime="2026-11-03 22:12:56.46213745 +0000 UTC m=+5078.093450926"
Nov 03 22:12:57.359689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:57.359654    5533 generic.go:356] "Generic (PLEG): container finished" podID="7273cdc7-2f31-403d-b4c8-3027674f87b4" containerID="b3473f77e8678869a736239e832ead8a6574323e3a68b59ffe9fa80606eb625e" exitCode=0
Nov 03 22:12:57.360195 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:57.359730    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh" event={"ID":"7273cdc7-2f31-403d-b4c8-3027674f87b4","Type":"ContainerDied","Data":"b3473f77e8678869a736239e832ead8a6574323e3a68b59ffe9fa80606eb625e"}
Nov 03 22:12:58.502979 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.502951    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:58.591113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.591083    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\") pod \"7273cdc7-2f31-403d-b4c8-3027674f87b4\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") "
Nov 03 22:12:58.591287 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.591130    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\") pod \"7273cdc7-2f31-403d-b4c8-3027674f87b4\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") "
Nov 03 22:12:58.591287 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.591155    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\" (UniqueName: \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\") pod \"7273cdc7-2f31-403d-b4c8-3027674f87b4\" (UID: \"7273cdc7-2f31-403d-b4c8-3027674f87b4\") "
Nov 03 22:12:58.591504 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.591478    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca" pod "7273cdc7-2f31-403d-b4c8-3027674f87b4" (UID: "7273cdc7-2f31-403d-b4c8-3027674f87b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:12:58.592665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.592613    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:58.593478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.593457    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca" pod "7273cdc7-2f31-403d-b4c8-3027674f87b4" (UID: "7273cdc7-2f31-403d-b4c8-3027674f87b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:12:58.593796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.593777    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl" pod "7273cdc7-2f31-403d-b4c8-3027674f87b4" (UID: "7273cdc7-2f31-403d-b4c8-3027674f87b4"). InnerVolumeSpecName "kube-api-access-pk5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:12:58.694024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.693996    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273cdc7-2f31-403d-b4c8-3027674f87b4-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:58.694024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:58.694020    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5jl\" (UniqueName: \"kubernetes.io/projected/7273cdc7-2f31-403d-b4c8-3027674f87b4-kube-api-access-pk5jl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:12:59.371959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:59.371932    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"
Nov 03 22:12:59.372122 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:59.371930    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh" event={"ID":"7273cdc7-2f31-403d-b4c8-3027674f87b4","Type":"ContainerDied","Data":"f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032"}
Nov 03 22:12:59.372122 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:12:59.372051    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2c136bc7660d91eff0dce1dea4081ac7a1184a0aea32bd98db051dbfd90f032"
Nov 03 22:13:02.195102 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:02.195058    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:13:15.641024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.640987    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"]
Nov 03 22:13:15.641482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.641363    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7273cdc7-2f31-403d-b4c8-3027674f87b4" containerName="revoke-keys"
Nov 03 22:13:15.641482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.641374    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7273cdc7-2f31-403d-b4c8-3027674f87b4" containerName="revoke-keys"
Nov 03 22:13:15.641482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.641425    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7273cdc7-2f31-403d-b4c8-3027674f87b4" containerName="revoke-keys"
Nov 03 22:13:15.644807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.644783    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.699970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.699940    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"]
Nov 03 22:13:15.743740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.743707    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:13:15.743875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.743773    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:13:15.747169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.747148    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.747231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.747180    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p4s\" (UniqueName: \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.747231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.747207    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.848059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.848031    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.848198 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.848064    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p4s\" (UniqueName: \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.848198 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.848136    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.848695 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.848674    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.850388 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.850368    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.905843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.905783    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p4s\" (UniqueName: \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\") pod \"maas-api-revoke-sub-0165d7d2-fkwvh\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:15.958108 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:15.958086    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:16.325024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:16.324994    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"]
Nov 03 22:13:16.327125 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:13:16.327048    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5226da_d475_41ae_9e6d_94c4b3bc435c.slice/crio-468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e WatchSource:0}: Error finding container 468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e: Status 404 returned error can't find the container with id 468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e
Nov 03 22:13:16.491615 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:16.491585    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh" event={"ID":"be5226da-d475-41ae-9e6d-94c4b3bc435c","Type":"ContainerStarted","Data":"468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e"}
Nov 03 22:13:17.495856 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:17.495823    5533 generic.go:356] "Generic (PLEG): container finished" podID="be5226da-d475-41ae-9e6d-94c4b3bc435c" containerID="5d5539b0dade39fb564d8d0b508451b1e8b50a3a0608841c20b3da264985118f" exitCode=0
Nov 03 22:13:17.496262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:17.495885    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh" event={"ID":"be5226da-d475-41ae-9e6d-94c4b3bc435c","Type":"ContainerDied","Data":"5d5539b0dade39fb564d8d0b508451b1e8b50a3a0608841c20b3da264985118f"}
Nov 03 22:13:18.090747 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.090711    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"]
Nov 03 22:13:18.097946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.097922    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34c9d3d9-jgtdl"]
Nov 03 22:13:18.492310 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.492277    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e781bf4-58fd-4653-8dc1-5daa300e12e5" path="/var/lib/kubelet/pods/1e781bf4-58fd-4653-8dc1-5daa300e12e5/volumes"
Nov 03 22:13:18.624495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.624471    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:18.680329 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.680307    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\" (UniqueName: \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\") pod \"be5226da-d475-41ae-9e6d-94c4b3bc435c\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") "
Nov 03 22:13:18.680455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.680369    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\") pod \"be5226da-d475-41ae-9e6d-94c4b3bc435c\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") "
Nov 03 22:13:18.680455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.680404    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\") pod \"be5226da-d475-41ae-9e6d-94c4b3bc435c\" (UID: \"be5226da-d475-41ae-9e6d-94c4b3bc435c\") "
Nov 03 22:13:18.680797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.680772    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca" pod "be5226da-d475-41ae-9e6d-94c4b3bc435c" (UID: "be5226da-d475-41ae-9e6d-94c4b3bc435c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:13:18.682351 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.682329    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca" pod "be5226da-d475-41ae-9e6d-94c4b3bc435c" (UID: "be5226da-d475-41ae-9e6d-94c4b3bc435c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:13:18.682448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.682429    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s" pod "be5226da-d475-41ae-9e6d-94c4b3bc435c" (UID: "be5226da-d475-41ae-9e6d-94c4b3bc435c"). InnerVolumeSpecName "kube-api-access-62p4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:13:18.781582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.781528    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p4s\" (UniqueName: \"kubernetes.io/projected/be5226da-d475-41ae-9e6d-94c4b3bc435c-kube-api-access-62p4s\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:18.781582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.781552    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:18.781582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:18.781561    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5226da-d475-41ae-9e6d-94c4b3bc435c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:19.260614 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:13:19.260578    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:13:19.447917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.447897    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:13:19.458548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.458528    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:13:19.496004 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.495987    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:13:19.506980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.506959    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"
Nov 03 22:13:19.506980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.506968    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh" event={"ID":"be5226da-d475-41ae-9e6d-94c4b3bc435c","Type":"ContainerDied","Data":"468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e"}
Nov 03 22:13:19.507164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.507014    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468ed56e28bdf2e4d6478c8c6adee343575e61053ada44fc084f093758ec557e"
Nov 03 22:13:19.507220 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:19.507203    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:13:22.356169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:22.356131    5533 scope.go:122] "RemoveContainer" containerID="3ec5a52115e47b5847b19c37e1270be0b38b1517f5fc8f26a6b0a256d7fb0f7f"
Nov 03 22:13:22.365161 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:22.365140    5533 scope.go:122] "RemoveContainer" containerID="f3436f49aa49288f1ce9f9d9255595c50473ec00cb689e925b87876d2ff50b0d"
Nov 03 22:13:22.373227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:22.373209    5533 scope.go:122] "RemoveContainer" containerID="860e39f791119fd5d9ad81c7c24e8d4c6280274591a8b1837dbcabd9e94a5b1f"
Nov 03 22:13:22.380839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:22.380807    5533 scope.go:122] "RemoveContainer" containerID="503bf974e5002c45820b9e9b5af3c07bd9d54aa6ca3cc991c782f8eb6361f66a"
Nov 03 22:13:27.472229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.472194    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"]
Nov 03 22:13:27.472614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.472555    5533 cpu_manager.go:414] "containerMap: removing container" podUID="be5226da-d475-41ae-9e6d-94c4b3bc435c" containerName="revoke-keys"
Nov 03 22:13:27.472614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.472566    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5226da-d475-41ae-9e6d-94c4b3bc435c" containerName="revoke-keys"
Nov 03 22:13:27.472857 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.472620    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5226da-d475-41ae-9e6d-94c4b3bc435c" containerName="revoke-keys"
Nov 03 22:13:27.477578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.477550    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.530854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.530822    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"]
Nov 03 22:13:27.548487 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.548463    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.548631 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.548515    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.548719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.548631    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnpb\" (UniqueName: \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.574690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.574662    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:13:27.574803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.574728    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:13:27.649097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.649067    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.649236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.649153    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.649236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.649229    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnpb\" (UniqueName: \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.649915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.649893    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.651438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.651420    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.706951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.706927    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnpb\" (UniqueName: \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\") pod \"maas-api-revoke-sub-6b19338f-vqjb9\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:27.790881 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:27.790833    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:28.155911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:28.155885    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"]
Nov 03 22:13:28.158364 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:13:28.158335    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb274855d_f6eb_4bff_9c00_8df0b1c37a68.slice/crio-d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48 WatchSource:0}: Error finding container d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48: Status 404 returned error can't find the container with id d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48
Nov 03 22:13:28.553382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:28.553354    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9" event={"ID":"b274855d-f6eb-4bff-9c00-8df0b1c37a68","Type":"ContainerStarted","Data":"faa593e98af66411d5d4ad173b39fcf6f4dff4af3980bbc8da3573ef6a0994e3"}
Nov 03 22:13:28.553715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:28.553407    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9" event={"ID":"b274855d-f6eb-4bff-9c00-8df0b1c37a68","Type":"ContainerStarted","Data":"d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48"}
Nov 03 22:13:28.666038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:28.665990    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9" podStartSLOduration=1.665977628 podStartE2EDuration="1.665977628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:28.662931992 +0000 UTC m=+5110.294245469" watchObservedRunningTime="2026-11-03 22:13:28.665977628 +0000 UTC m=+5110.297291104"
Nov 03 22:13:29.560587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:29.560552    5533 generic.go:356] "Generic (PLEG): container finished" podID="b274855d-f6eb-4bff-9c00-8df0b1c37a68" containerID="faa593e98af66411d5d4ad173b39fcf6f4dff4af3980bbc8da3573ef6a0994e3" exitCode=0
Nov 03 22:13:29.561004 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:29.560653    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9" event={"ID":"b274855d-f6eb-4bff-9c00-8df0b1c37a68","Type":"ContainerDied","Data":"faa593e98af66411d5d4ad173b39fcf6f4dff4af3980bbc8da3573ef6a0994e3"}
Nov 03 22:13:30.697325 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.697297    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:30.774557 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.774532    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\") pod \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") "
Nov 03 22:13:30.774697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.774585    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\" (UniqueName: \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\") pod \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") "
Nov 03 22:13:30.774697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.774616    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\") pod \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\" (UID: \"b274855d-f6eb-4bff-9c00-8df0b1c37a68\") "
Nov 03 22:13:30.775030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.775008    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca" pod "b274855d-f6eb-4bff-9c00-8df0b1c37a68" (UID: "b274855d-f6eb-4bff-9c00-8df0b1c37a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:13:30.776532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.776507    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca" pod "b274855d-f6eb-4bff-9c00-8df0b1c37a68" (UID: "b274855d-f6eb-4bff-9c00-8df0b1c37a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:13:30.776620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.776595    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb" pod "b274855d-f6eb-4bff-9c00-8df0b1c37a68" (UID: "b274855d-f6eb-4bff-9c00-8df0b1c37a68"). InnerVolumeSpecName "kube-api-access-prnpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:13:30.876200 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.876138    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:30.876200 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.876163    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prnpb\" (UniqueName: \"kubernetes.io/projected/b274855d-f6eb-4bff-9c00-8df0b1c37a68-kube-api-access-prnpb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:30.876200 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:30.876175    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b274855d-f6eb-4bff-9c00-8df0b1c37a68-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:13:31.571070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:31.571039    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"
Nov 03 22:13:31.571224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:31.571080    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9" event={"ID":"b274855d-f6eb-4bff-9c00-8df0b1c37a68","Type":"ContainerDied","Data":"d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48"}
Nov 03 22:13:31.571224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:31.571129    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d640344bf1409cf22583f2356ef823f8662eb4cf1dc17c69d6833383eb9fae48"
Nov 03 22:13:35.045325 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:35.045282    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"]
Nov 03 22:13:35.091749 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:35.091721    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d053e827-nsvqq"]
Nov 03 22:13:36.492622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:36.492580    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e" path="/var/lib/kubelet/pods/f2cdf215-1c68-4bcf-a11d-9c19f78b9c1e/volumes"
Nov 03 22:13:40.293741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:40.293707    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:13:43.434702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.434663    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:13:43.435188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.435108    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b274855d-f6eb-4bff-9c00-8df0b1c37a68" containerName="revoke-keys"
Nov 03 22:13:43.435188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.435124    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b274855d-f6eb-4bff-9c00-8df0b1c37a68" containerName="revoke-keys"
Nov 03 22:13:43.435188 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.435187    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b274855d-f6eb-4bff-9c00-8df0b1c37a68" containerName="revoke-keys"
Nov 03 22:13:43.438430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.438406    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.534436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.534407    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-afdcb5ab-kserve-self-signed-certs\""
Nov 03 22:13:43.536778 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.536754    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:13:43.629026 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.628993    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.629210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.629043    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.629210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.629121    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.629210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.629190    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.629332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.629285    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6pp\" (UniqueName: \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.629332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.629317    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730492    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6pp\" (UniqueName: \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730529    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730566    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730602    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730623    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.730796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730668    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.731027 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.730978    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.731105 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.731032    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.731162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.731100    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.732988 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.732965    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.733240 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.733221    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:43.788468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:43.788441    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6pp\" (UniqueName: \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\") pod \"e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") " pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:44.053215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:44.053160    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:44.417336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:44.417308    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:13:44.418779 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:13:44.418748    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5178c4e6_66cb_411f_bb07_cdf40fb56219.slice/crio-c4a3a507f272de7171d67f43988e512c905fac404381a715516c8fa874f2d08f WatchSource:0}: Error finding container c4a3a507f272de7171d67f43988e512c905fac404381a715516c8fa874f2d08f: Status 404 returned error can't find the container with id c4a3a507f272de7171d67f43988e512c905fac404381a715516c8fa874f2d08f
Nov 03 22:13:44.658978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:44.658940    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" event={"ID":"5178c4e6-66cb-411f-bb07-cdf40fb56219","Type":"ContainerStarted","Data":"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"}
Nov 03 22:13:44.659378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:44.658994    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" event={"ID":"5178c4e6-66cb-411f-bb07-cdf40fb56219","Type":"ContainerStarted","Data":"c4a3a507f272de7171d67f43988e512c905fac404381a715516c8fa874f2d08f"}
Nov 03 22:13:44.775372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:44.775309    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" podStartSLOduration=1.7752901410000002 podStartE2EDuration="1.775290141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:44.773464332 +0000 UTC m=+5126.404777812" watchObservedRunningTime="2026-11-03 22:13:44.775290141 +0000 UTC m=+5126.406603621"
Nov 03 22:13:45.053662 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:45.053569    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:49.332885 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:13:49.332835    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:13:54.054257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:54.054226    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:54.066481 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:54.066457    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:54.762655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:54.762600    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:13:56.497105 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:13:56.497076    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:14:08.045701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:08.045615    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"]
Nov 03 22:14:08.092343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:08.092311    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9061f323-nhbgs"]
Nov 03 22:14:08.491981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:08.491946    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f293a7-5b5e-4585-bdbb-f51d3dbc0569" path="/var/lib/kubelet/pods/f2f293a7-5b5e-4585-bdbb-f51d3dbc0569/volumes"
Nov 03 22:14:19.260450 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:14:19.260414    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:14:22.413621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:22.413580    5533 scope.go:122] "RemoveContainer" containerID="f755fc5d6d5b1a52c9a3b0315efafcc4515bac3bfedc6d3b2020bd0a68e067d4"
Nov 03 22:14:22.422601 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:22.422582    5533 scope.go:122] "RemoveContainer" containerID="051102faccd5c59c9318d4b290d3d456b64e9a8f9f69b321c0ee1e9cacca803e"
Nov 03 22:14:22.430383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:22.430358    5533 scope.go:122] "RemoveContainer" containerID="747a8e514a724a3b21c9abd21a18b4ced1d10ad6a12c88d7cda44c2c3a605a3a"
Nov 03 22:14:26.051025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:26.050986    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"]
Nov 03 22:14:26.098587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:26.098552    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad411417-6vtqw"]
Nov 03 22:14:26.406692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:26.406611    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:14:26.492404 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:26.492369    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b88ad4-4816-4515-9aa2-ba2e3625b760" path="/var/lib/kubelet/pods/96b88ad4-4816-4515-9aa2-ba2e3625b760/volumes"
Nov 03 22:14:37.577519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.577486    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"]
Nov 03 22:14:37.580816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.580797    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.630298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.630275    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:14:37.664920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.664897    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cwq\" (UniqueName: \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.665031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.664944    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.665076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.665051    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.677067 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.677043    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:14:37.679203 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.679181    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"]
Nov 03 22:14:37.765708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.765680    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.765808 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.765717    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cwq\" (UniqueName: \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.765808 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.765759    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.766418 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.766401    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.768003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.767986    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.823087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.823060    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cwq\" (UniqueName: \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\") pod \"maas-api-revoke-sub-111335a5-nsz4f\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:37.893768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:37.893716    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:38.261397 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:38.261368    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"]
Nov 03 22:14:38.263126 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:14:38.263099    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc86db23_88e6_434d_bea7_90270aaed199.slice/crio-bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529 WatchSource:0}: Error finding container bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529: Status 404 returned error can't find the container with id bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529
Nov 03 22:14:38.998692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:38.998653    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f" event={"ID":"bc86db23-88e6-434d-bea7-90270aaed199","Type":"ContainerStarted","Data":"e5f686b4b65debdbef92b45def41cdb388945e2a57e0e9c630179d056913dda7"}
Nov 03 22:14:38.999152 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:38.998717    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f" event={"ID":"bc86db23-88e6-434d-bea7-90270aaed199","Type":"ContainerStarted","Data":"bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529"}
Nov 03 22:14:39.109263 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:39.109211    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f" podStartSLOduration=2.10919745 podStartE2EDuration="2.10919745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:39.108062916 +0000 UTC m=+5180.739376387" watchObservedRunningTime="2026-11-03 22:14:39.10919745 +0000 UTC m=+5180.740510929"
Nov 03 22:14:40.005566 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:40.005533    5533 generic.go:356] "Generic (PLEG): container finished" podID="bc86db23-88e6-434d-bea7-90270aaed199" containerID="e5f686b4b65debdbef92b45def41cdb388945e2a57e0e9c630179d056913dda7" exitCode=0
Nov 03 22:14:40.006026 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:40.005615    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f" event={"ID":"bc86db23-88e6-434d-bea7-90270aaed199","Type":"ContainerDied","Data":"e5f686b4b65debdbef92b45def41cdb388945e2a57e0e9c630179d056913dda7"}
Nov 03 22:14:41.143202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.143174    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:41.297764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.297698    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\") pod \"bc86db23-88e6-434d-bea7-90270aaed199\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") "
Nov 03 22:14:41.297764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.297736    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\" (UniqueName: \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\") pod \"bc86db23-88e6-434d-bea7-90270aaed199\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") "
Nov 03 22:14:41.297934 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.297819    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\") pod \"bc86db23-88e6-434d-bea7-90270aaed199\" (UID: \"bc86db23-88e6-434d-bea7-90270aaed199\") "
Nov 03 22:14:41.298059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.298037    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca" pod "bc86db23-88e6-434d-bea7-90270aaed199" (UID: "bc86db23-88e6-434d-bea7-90270aaed199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:41.299829 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.299798    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca" pod "bc86db23-88e6-434d-bea7-90270aaed199" (UID: "bc86db23-88e6-434d-bea7-90270aaed199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:41.299937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.299895    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq" pod "bc86db23-88e6-434d-bea7-90270aaed199" (UID: "bc86db23-88e6-434d-bea7-90270aaed199"). InnerVolumeSpecName "kube-api-access-25cwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:41.398417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.398392    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:14:41.398417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.398415    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc86db23-88e6-434d-bea7-90270aaed199-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:14:41.398539 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:41.398426    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25cwq\" (UniqueName: \"kubernetes.io/projected/bc86db23-88e6-434d-bea7-90270aaed199-kube-api-access-25cwq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:14:42.017509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:42.017479    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"
Nov 03 22:14:42.017718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:42.017474    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f" event={"ID":"bc86db23-88e6-434d-bea7-90270aaed199","Type":"ContainerDied","Data":"bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529"}
Nov 03 22:14:42.017718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:42.017609    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1e222426cfa817f99ce84d3f14f326ca8bc44dbfd884a52fa084213ab9e529"
Nov 03 22:14:49.260203 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:14:49.260169    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:14:50.595419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:50.595389    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:14:50.697459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:50.697424    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:14:56.262126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.262088    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"]
Nov 03 22:14:56.262712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.262663    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bc86db23-88e6-434d-bea7-90270aaed199" containerName="revoke-keys"
Nov 03 22:14:56.262712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.262685    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc86db23-88e6-434d-bea7-90270aaed199" containerName="revoke-keys"
Nov 03 22:14:56.262853 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.262794    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc86db23-88e6-434d-bea7-90270aaed199" containerName="revoke-keys"
Nov 03 22:14:56.266179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.266155    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.328352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.328318    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.328491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.328454    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wdl\" (UniqueName: \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.328547 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.328519    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.370509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.370467    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"]
Nov 03 22:14:56.370929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.370905    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:14:56.371120 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.371105    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:14:56.430154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.430105    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wdl\" (UniqueName: \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.430352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.430176    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.430352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.430250    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.431037 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.431002    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.432994 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.432946    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.494740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.493850    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wdl\" (UniqueName: \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\") pod \"maas-api-revoke-sub-75b4649d-9wsr5\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.582618 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.582544    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:56.972791 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:56.972699    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"]
Nov 03 22:14:57.102577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:57.102542    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5" event={"ID":"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2","Type":"ContainerStarted","Data":"b70925cc94dab11f639e582ce7bd2ad36e1671fcb5d7684a3de37d06417906ca"}
Nov 03 22:14:58.138114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:58.138086    5533 generic.go:356] "Generic (PLEG): container finished" podID="4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" containerID="2c388c9725d2b3367c24529d9dabba8bfc4b9cdcbe967f1bfa4a5cbca0f44368" exitCode=0
Nov 03 22:14:58.138510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:58.138149    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5" event={"ID":"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2","Type":"ContainerDied","Data":"2c388c9725d2b3367c24529d9dabba8bfc4b9cdcbe967f1bfa4a5cbca0f44368"}
Nov 03 22:14:59.262463 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.262438    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:14:59.366353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.366325    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\") pod \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") "
Nov 03 22:14:59.366491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.366365    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\" (UniqueName: \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\") pod \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") "
Nov 03 22:14:59.366491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.366472    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\") pod \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\" (UID: \"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2\") "
Nov 03 22:14:59.366848 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.366818    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca" pod "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" (UID: "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:59.368551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.368521    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl" pod "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" (UID: "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2"). InnerVolumeSpecName "kube-api-access-l4wdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:59.368551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.368525    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca" pod "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" (UID: "4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:59.467843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.467818    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:14:59.467843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.467844    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:14:59.467996 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:14:59.467854    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wdl\" (UniqueName: \"kubernetes.io/projected/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2-kube-api-access-l4wdl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:00.152126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.152089    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5" event={"ID":"4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2","Type":"ContainerDied","Data":"b70925cc94dab11f639e582ce7bd2ad36e1671fcb5d7684a3de37d06417906ca"}
Nov 03 22:15:00.152330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.152144    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b70925cc94dab11f639e582ce7bd2ad36e1671fcb5d7684a3de37d06417906ca"
Nov 03 22:15:00.152330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.152200    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"
Nov 03 22:15:00.153338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.153312    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:00.153738 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.153721    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" containerName="revoke-keys"
Nov 03 22:15:00.153823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.153741    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" containerName="revoke-keys"
Nov 03 22:15:00.153885 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.153875    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" containerName="revoke-keys"
Nov 03 22:15:00.157085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.157063    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.254609 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.254563    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:15:00.256792 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.256764    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:00.274086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.274052    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.274485 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.274169    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jz8l\" (UniqueName: \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.375321 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.375293    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.375443 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.375405    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jz8l\" (UniqueName: \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.375982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.375962    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.433332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.433308    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jz8l\" (UniqueName: \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\") pod \"maas-api-key-cleanup-29895735-5f7dd\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.473873 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.473853    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:00.838569 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:00.838545    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:00.840447 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:15:00.840416    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacc5cb5_3b9b_48f9_8113_421c0bdb9323.slice/crio-6fed08e9d8b1cc9c9827ef99452859a3f9fc3ea47b6f9aefdbafe96c0b384baf WatchSource:0}: Error finding container 6fed08e9d8b1cc9c9827ef99452859a3f9fc3ea47b6f9aefdbafe96c0b384baf: Status 404 returned error can't find the container with id 6fed08e9d8b1cc9c9827ef99452859a3f9fc3ea47b6f9aefdbafe96c0b384baf
Nov 03 22:15:01.159280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:01.159254    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerStarted","Data":"6fed08e9d8b1cc9c9827ef99452859a3f9fc3ea47b6f9aefdbafe96c0b384baf"}
Nov 03 22:15:02.106951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:02.106917    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:15:02.166417 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:02.166386    5533 generic.go:356] "Generic (PLEG): container finished" podID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerID="db753726c837a6557ae6790d0a187610edd0edfe1c03d823cce1dbbe7b1457fe" exitCode=60
Nov 03 22:15:02.166563 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:02.166463    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerDied","Data":"db753726c837a6557ae6790d0a187610edd0edfe1c03d823cce1dbbe7b1457fe"}
Nov 03 22:15:02.166629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:02.166614    5533 scope.go:122] "RemoveContainer" containerID="db753726c837a6557ae6790d0a187610edd0edfe1c03d823cce1dbbe7b1457fe"
Nov 03 22:15:03.173377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:03.173348    5533 generic.go:356] "Generic (PLEG): container finished" podID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerID="2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc" exitCode=60
Nov 03 22:15:03.173783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:03.173432    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerDied","Data":"2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc"}
Nov 03 22:15:03.173783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:03.173487    5533 scope.go:122] "RemoveContainer" containerID="db753726c837a6557ae6790d0a187610edd0edfe1c03d823cce1dbbe7b1457fe"
Nov 03 22:15:03.173783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:03.173721    5533 scope.go:122] "RemoveContainer" containerID="2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc"
Nov 03 22:15:03.173979 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:03.173955    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-5f7dd_odh-ai-gateway-infra(bacc5cb5-3b9b-48f9-8113-421c0bdb9323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323"
Nov 03 22:15:04.187607 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:04.187581    5533 scope.go:122] "RemoveContainer" containerID="2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc"
Nov 03 22:15:04.188009 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:04.187804    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-5f7dd_odh-ai-gateway-infra(bacc5cb5-3b9b-48f9-8113-421c0bdb9323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323"
Nov 03 22:15:05.717695 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:05.717656    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerName="main" containerID="cri-o://4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745" gracePeriod=45
Nov 03 22:15:05.751661 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:05.751609    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerName="main" probeResult="failure" output="Get \"https://10.133.1.103:8000/ready\": dial tcp 10.133.1.103:8000: connect: connection refused"
Nov 03 22:15:05.965586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:05.965559    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:15:06.024031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.023951    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\" (UniqueName: \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024022    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024103    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\" (UniqueName: \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024170    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024196    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024221    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\") pod \"5178c4e6-66cb-411f-bb07-cdf40fb56219\" (UID: \"5178c4e6-66cb-411f-bb07-cdf40fb56219\") "
Nov 03 22:15:06.024470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024447    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:06.024517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024462    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:06.024589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024570    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.024678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024580    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:06.024678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.024597    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.026247 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.026227    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:15:06.026344 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.026326    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "kube-api-access-rr6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:06.026344 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.026333    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs" pod "5178c4e6-66cb-411f-bb07-cdf40fb56219" (UID: "5178c4e6-66cb-411f-bb07-cdf40fb56219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:15:06.126058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.126023    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6pp\" (UniqueName: \"kubernetes.io/projected/5178c4e6-66cb-411f-bb07-cdf40fb56219-kube-api-access-rr6pp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.126058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.126054    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.126242 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.126066    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5178c4e6-66cb-411f-bb07-cdf40fb56219-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.126242 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.126078    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5178c4e6-66cb-411f-bb07-cdf40fb56219-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:06.198897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.198869    5533 generic.go:356] "Generic (PLEG): container finished" podID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerID="4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745" exitCode=0
Nov 03 22:15:06.199090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.198916    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"
Nov 03 22:15:06.199090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.198954    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" event={"ID":"5178c4e6-66cb-411f-bb07-cdf40fb56219","Type":"ContainerDied","Data":"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"}
Nov 03 22:15:06.199090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.199001    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv" event={"ID":"5178c4e6-66cb-411f-bb07-cdf40fb56219","Type":"ContainerDied","Data":"c4a3a507f272de7171d67f43988e512c905fac404381a715516c8fa874f2d08f"}
Nov 03 22:15:06.199090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.199024    5533 scope.go:122] "RemoveContainer" containerID="4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"
Nov 03 22:15:06.209426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.209411    5533 scope.go:122] "RemoveContainer" containerID="4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"
Nov 03 22:15:06.209679 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:06.209662    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745\": container with ID starting with 4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745 not found: ID does not exist" containerID="4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"
Nov 03 22:15:06.209732 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.209686    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745"} err="failed to get container status \"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745\": rpc error: code = NotFound desc = could not find container \"4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745\": container with ID starting with 4e3f6caa9435256079c11bf135529a85f45b415f3f7df082b04c11467f85e745 not found: ID does not exist"
Nov 03 22:15:06.355581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.355517    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:15:06.359538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.359510    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-afdcb5ab-kserve-8bd95dc99-wx9vv"]
Nov 03 22:15:06.491942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:06.491909    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" path="/var/lib/kubelet/pods/5178c4e6-66cb-411f-bb07-cdf40fb56219/volumes"
Nov 03 22:15:09.422495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.422463    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:15:09.422917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.422869    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerName="main"
Nov 03 22:15:09.422917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.422882    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerName="main"
Nov 03 22:15:09.422995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.422943    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5178c4e6-66cb-411f-bb07-cdf40fb56219" containerName="main"
Nov 03 22:15:09.427304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.427287    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.476867 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.476829    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-60736a-openshift-default-dockercfg-6q8vb\""
Nov 03 22:15:09.526264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.526234    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:15:09.553438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553406    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.553610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553457    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.553610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553531    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.553781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553767    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w86r\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.553838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553826    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.553880 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.553857    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.554020 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.554001    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.554092 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.554029    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.554140 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.554113    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.654959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.654924    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.654968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w86r\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655019    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655066    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655149 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655131    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655160    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655234    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655287    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655314    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655515    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655530    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655762 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655630    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.655826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.655803    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.656110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.656092    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.657388 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.657367    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.657543 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.657522    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.712869 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.712802    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.713030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.713005    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w86r\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\") pod \"e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") " pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:09.740224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:09.740193    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:10.163963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:10.163936    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:15:10.164842 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:15:10.164811    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a1ce01_c0d2_43c6_a45c_4f8f0a57d7c3.slice/crio-66501cbc8648c84a0e616273711b41be8ab047f5870edfebdb056d7738a6ac36 WatchSource:0}: Error finding container 66501cbc8648c84a0e616273711b41be8ab047f5870edfebdb056d7738a6ac36: Status 404 returned error can't find the container with id 66501cbc8648c84a0e616273711b41be8ab047f5870edfebdb056d7738a6ac36
Nov 03 22:15:10.166824 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:10.166794    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:10.166906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:10.166858    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:10.166906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:10.166888    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:10.221647 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:10.221606    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerStarted","Data":"66501cbc8648c84a0e616273711b41be8ab047f5870edfebdb056d7738a6ac36"}
Nov 03 22:15:11.227984 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:11.227946    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerStarted","Data":"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"}
Nov 03 22:15:11.388141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:11.388095    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podStartSLOduration=2.388080744 podStartE2EDuration="2.388080744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:11.387293701 +0000 UTC m=+5213.018607177" watchObservedRunningTime="2026-11-03 22:15:11.388080744 +0000 UTC m=+5213.019394221"
Nov 03 22:15:11.740594 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:11.740558    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:11.745741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:11.745716    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:12.233786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:12.233753    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:12.234702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:12.234675    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:15.486440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:15.486400    5533 scope.go:122] "RemoveContainer" containerID="2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc"
Nov 03 22:15:16.262412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:16.262378    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerStarted","Data":"09ec688fadf26e0d8fe90179733fb0b6a231e042ead926f2a66cc51a0279c84e"}
Nov 03 22:15:16.374082 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:16.374018    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" podStartSLOduration=16.374005566 podStartE2EDuration="16.374005566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:16.372708747 +0000 UTC m=+5218.004022223" watchObservedRunningTime="2026-11-03 22:15:16.374005566 +0000 UTC m=+5218.005319331"
Nov 03 22:15:17.270047 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:17.270017    5533 generic.go:356] "Generic (PLEG): container finished" podID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerID="09ec688fadf26e0d8fe90179733fb0b6a231e042ead926f2a66cc51a0279c84e" exitCode=60
Nov 03 22:15:17.270537 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:17.270095    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerDied","Data":"09ec688fadf26e0d8fe90179733fb0b6a231e042ead926f2a66cc51a0279c84e"}
Nov 03 22:15:17.270537 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:17.270170    5533 scope.go:122] "RemoveContainer" containerID="2d554e640d52d067a9315a9baf3f52c38220dd5c2edc6f00e8be98e9121964bc"
Nov 03 22:15:17.270537 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:17.270505    5533 scope.go:122] "RemoveContainer" containerID="09ec688fadf26e0d8fe90179733fb0b6a231e042ead926f2a66cc51a0279c84e"
Nov 03 22:15:17.270769 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:17.270753    5533 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895735-5f7dd_odh-ai-gateway-infra(bacc5cb5-3b9b-48f9-8113-421c0bdb9323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323"
Nov 03 22:15:17.388546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:17.388524    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:18.175335 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.175304    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:15:18.180028 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.180009    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.228927 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.228899    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-60736a\""
Nov 03 22:15:18.256429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.256401    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tprs\" (UniqueName: \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.256585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.256475    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.286681 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.279318    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bed1ddc8-dockercfg-j4wpt\""
Nov 03 22:15:18.306710 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.294210    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:15:18.321181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.320335    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:15:18.326761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.326722    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.357587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.357523    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k8ch\" (UniqueName: \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.357830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.357659    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tprs\" (UniqueName: \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.357830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.357735    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.357830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.357798    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.358788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.358545    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.380059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.379965    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:15:18.430289 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.430230    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tprs\" (UniqueName: \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\") pod \"payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.447205 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.447182    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:18.458644 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.458603    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.458756 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.458738    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k8ch\" (UniqueName: \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.459534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.459511    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.492610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.492586    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:18.516799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.516774    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k8ch\" (UniqueName: \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\") pod \"payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.562194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.562157    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\" (UniqueName: \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\") pod \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") "
Nov 03 22:15:18.562297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.562226    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\") pod \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\" (UID: \"bacc5cb5-3b9b-48f9-8113-421c0bdb9323\") "
Nov 03 22:15:18.563596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.563573    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca" pod "bacc5cb5-3b9b-48f9-8113-421c0bdb9323" (UID: "bacc5cb5-3b9b-48f9-8113-421c0bdb9323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:18.566010 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.565984    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l" pod "bacc5cb5-3b9b-48f9-8113-421c0bdb9323" (UID: "bacc5cb5-3b9b-48f9-8113-421c0bdb9323"). InnerVolumeSpecName "kube-api-access-9jz8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:18.641359 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.641333    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:18.663390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.663366    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jz8l\" (UniqueName: \"kubernetes.io/projected/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-kube-api-access-9jz8l\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.663390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.663389    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacc5cb5-3b9b-48f9-8113-421c0bdb9323-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.861974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:18.861911    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:15:18.862104 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:15:18.862065    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1e6bba_caa2_4bed_a303_b36f74e24a05.slice/crio-aabddbc73bdf063a54e4f5f6069ba2b06eb53e5de343d9f35651ebb77e365108 WatchSource:0}: Error finding container aabddbc73bdf063a54e4f5f6069ba2b06eb53e5de343d9f35651ebb77e365108: Status 404 returned error can't find the container with id aabddbc73bdf063a54e4f5f6069ba2b06eb53e5de343d9f35651ebb77e365108
Nov 03 22:15:19.013182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.013158    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:15:19.014851 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:15:19.014825    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a62e52_84a9_4406_b26d_6bdf195697e3.slice/crio-bb89ea954a8e001bbf92c034b66601f5d62b139f6cb96f5bff7f6349c538633b WatchSource:0}: Error finding container bb89ea954a8e001bbf92c034b66601f5d62b139f6cb96f5bff7f6349c538633b: Status 404 returned error can't find the container with id bb89ea954a8e001bbf92c034b66601f5d62b139f6cb96f5bff7f6349c538633b
Nov 03 22:15:19.258984 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:19.258949    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:15:19.289499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.289424    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" event={"ID":"51a62e52-84a9-4406-b26d-6bdf195697e3","Type":"ContainerStarted","Data":"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"}
Nov 03 22:15:19.289878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.289521    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" event={"ID":"51a62e52-84a9-4406-b26d-6bdf195697e3","Type":"ContainerStarted","Data":"bb89ea954a8e001bbf92c034b66601f5d62b139f6cb96f5bff7f6349c538633b"}
Nov 03 22:15:19.289878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.289553    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:19.290690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.290666    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd" event={"ID":"bacc5cb5-3b9b-48f9-8113-421c0bdb9323","Type":"ContainerDied","Data":"6fed08e9d8b1cc9c9827ef99452859a3f9fc3ea47b6f9aefdbafe96c0b384baf"}
Nov 03 22:15:19.290690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.290684    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"
Nov 03 22:15:19.290867 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.290715    5533 scope.go:122] "RemoveContainer" containerID="09ec688fadf26e0d8fe90179733fb0b6a231e042ead926f2a66cc51a0279c84e"
Nov 03 22:15:19.292421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.292394    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" event={"ID":"5b1e6bba-caa2-4bed-a303-b36f74e24a05","Type":"ContainerStarted","Data":"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"}
Nov 03 22:15:19.292534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.292450    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:19.292534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.292475    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" event={"ID":"5b1e6bba-caa2-4bed-a303-b36f74e24a05","Type":"ContainerStarted","Data":"aabddbc73bdf063a54e4f5f6069ba2b06eb53e5de343d9f35651ebb77e365108"}
Nov 03 22:15:19.399957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.399911    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" podStartSLOduration=1.399900284 podStartE2EDuration="1.399900284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:19.398791952 +0000 UTC m=+5221.030105442" watchObservedRunningTime="2026-11-03 22:15:19.399900284 +0000 UTC m=+5221.031213760"
Nov 03 22:15:19.510518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.510479    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" podStartSLOduration=1.5104669689999999 podStartE2EDuration="1.510466969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:19.507917718 +0000 UTC m=+5221.139231195" watchObservedRunningTime="2026-11-03 22:15:19.510466969 +0000 UTC m=+5221.141780465"
Nov 03 22:15:19.612439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.612359    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:19.616100 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:19.616069    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895735-5f7dd"]
Nov 03 22:15:20.492361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:20.492329    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" path="/var/lib/kubelet/pods/bacc5cb5-3b9b-48f9-8113-421c0bdb9323/volumes"
Nov 03 22:15:22.458781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:22.458749    5533 scope.go:122] "RemoveContainer" containerID="36ea46baf9bfec8fffe53d6079acb767d7c2b2ad6a0a75203322e2860c9f0e9c"
Nov 03 22:15:22.467591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:22.467571    5533 scope.go:122] "RemoveContainer" containerID="203204d1b60e39f8d2dc4437eb681c9e4dd09e59f1f8a0160cbce62a7c5d9c76"
Nov 03 22:15:22.475215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:22.475191    5533 scope.go:122] "RemoveContainer" containerID="171668d797123744d43c8aa675de38b6e1667f2d97153b7c5cc465f8316d0f03"
Nov 03 22:15:29.166510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:29.166427    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:15:30.303033 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:30.302999    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:15:30.305412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:30.303095    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:15:38.423339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.423302    5533 generic.go:356] "Generic (PLEG): container finished" podID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerID="31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417" exitCode=0
Nov 03 22:15:38.423791 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.423387    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerDied","Data":"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"}
Nov 03 22:15:38.423791 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.423786    5533 scope.go:122] "RemoveContainer" containerID="31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"
Nov 03 22:15:38.424947 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.424922    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:38.425013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.424989    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:38.425051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:38.425019    5533 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 22:15:39.431383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:39.431340    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerStarted","Data":"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"}
Nov 03 22:15:39.740412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:39.740326    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:39.741420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:39.741397    5533 patch_prober.go:29] interesting pod/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Nov 03 22:15:39.741477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:39.741447    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Nov 03 22:15:40.740548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:40.740511    5533 patch_prober.go:29] interesting pod/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Nov 03 22:15:40.740975 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:40.740568    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Nov 03 22:15:41.741153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:41.741121    5533 patch_prober.go:29] interesting pod/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Nov 03 22:15:41.741562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:41.741174    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Nov 03 22:15:42.233798 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:42.233768    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:42.740987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:42.740954    5533 patch_prober.go:29] interesting pod/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Nov 03 22:15:42.741190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:42.741011    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Nov 03 22:15:43.741045 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:43.741005    5533 patch_prober.go:29] interesting pod/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Nov 03 22:15:43.741211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:43.741089    5533 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Nov 03 22:15:44.745295 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:44.745266    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:44.746279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:44.746259    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:15:47.426828 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:15:47.426785    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:15:49.261847 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:15:49.261812    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:16:07.905221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:07.905183    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:16:19.260255 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:16:19.260210    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:16:22.812447 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812413    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"]
Nov 03 22:16:22.812843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812805    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.812843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812816    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.812923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812845    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.812923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812851    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.812923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812910    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.812923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.812919    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:16:22.815061 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.815037    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:22.864164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.864139    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:16:22.910886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.910863    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:16:22.913123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.913098    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"]
Nov 03 22:16:22.974524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.974489    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxw2\" (UniqueName: \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:22.974672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.974551    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:22.974730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:22.974668    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.075821 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.075746    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxw2\" (UniqueName: \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.075821 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.075801    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.076008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.075845    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.076496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.076476    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.078406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.078379    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.133850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.133820    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxw2\" (UniqueName: \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\") pod \"maas-api-revoke-sub-9164cdf1-rkrff\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.426392 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.426363    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:23.791590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:23.791566    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"]
Nov 03 22:16:23.792839 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:16:23.792809    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1decd0f1_6b66_4777_984a_72662e3f2d99.slice/crio-d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4 WatchSource:0}: Error finding container d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4: Status 404 returned error can't find the container with id d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4
Nov 03 22:16:24.674508 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:24.674469    5533 generic.go:356] "Generic (PLEG): container finished" podID="1decd0f1-6b66-4777-984a-72662e3f2d99" containerID="7398cffbc1f9e404524e0c7a7a09390365edc8178654fbbab4cef8703d81bb86" exitCode=0
Nov 03 22:16:24.675041 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:24.674548    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff" event={"ID":"1decd0f1-6b66-4777-984a-72662e3f2d99","Type":"ContainerDied","Data":"7398cffbc1f9e404524e0c7a7a09390365edc8178654fbbab4cef8703d81bb86"}
Nov 03 22:16:24.675041 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:24.674598    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff" event={"ID":"1decd0f1-6b66-4777-984a-72662e3f2d99","Type":"ContainerStarted","Data":"d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4"}
Nov 03 22:16:25.806434 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.806409    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:25.904787 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.904760    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\" (UniqueName: \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\") pod \"1decd0f1-6b66-4777-984a-72662e3f2d99\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") "
Nov 03 22:16:25.904910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.904818    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\") pod \"1decd0f1-6b66-4777-984a-72662e3f2d99\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") "
Nov 03 22:16:25.904910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.904868    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\") pod \"1decd0f1-6b66-4777-984a-72662e3f2d99\" (UID: \"1decd0f1-6b66-4777-984a-72662e3f2d99\") "
Nov 03 22:16:25.905253 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.905230    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca" pod "1decd0f1-6b66-4777-984a-72662e3f2d99" (UID: "1decd0f1-6b66-4777-984a-72662e3f2d99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:16:25.906830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.906808    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca" pod "1decd0f1-6b66-4777-984a-72662e3f2d99" (UID: "1decd0f1-6b66-4777-984a-72662e3f2d99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:16:25.906923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:25.906841    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2" pod "1decd0f1-6b66-4777-984a-72662e3f2d99" (UID: "1decd0f1-6b66-4777-984a-72662e3f2d99"). InnerVolumeSpecName "kube-api-access-4dxw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:26.005890 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.005815    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:26.005890 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.005843    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1decd0f1-6b66-4777-984a-72662e3f2d99-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:26.005890 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.005853    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxw2\" (UniqueName: \"kubernetes.io/projected/1decd0f1-6b66-4777-984a-72662e3f2d99-kube-api-access-4dxw2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:26.689469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.689434    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"
Nov 03 22:16:26.689469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.689451    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff" event={"ID":"1decd0f1-6b66-4777-984a-72662e3f2d99","Type":"ContainerDied","Data":"d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4"}
Nov 03 22:16:26.689702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:26.689503    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d05e342aa70241fc6f97deb79a8245ac234e48254b757268e484fc6306cbe4"
Nov 03 22:16:35.875452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:35.875417    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:16:38.045291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:38.045254    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"]
Nov 03 22:16:38.091769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:38.091741    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rwsf8"]
Nov 03 22:16:38.491918 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:38.491887    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e76e4d6-e2cd-416a-9146-b3d09dbf5c37" path="/var/lib/kubelet/pods/0e76e4d6-e2cd-416a-9146-b3d09dbf5c37/volumes"
Nov 03 22:16:41.121849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:41.121813    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:16:41.228136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:41.228098    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:16:45.996372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:45.996342    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:16:46.122896 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.122850    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" podUID="51a62e52-84a9-4406-b26d-6bdf195697e3" containerName="payload-processing" containerID="cri-o://9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e" gracePeriod=25
Nov 03 22:16:46.229099 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.229061    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" podUID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" containerName="payload-pre-processing" containerID="cri-o://c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732" gracePeriod=25
Nov 03 22:16:46.379043 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.379016    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:16:46.466679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.466654    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:16:46.487486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.487452    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\") pod \"51a62e52-84a9-4406-b26d-6bdf195697e3\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") "
Nov 03 22:16:46.487661 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.487505    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\" (UniqueName: \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\") pod \"51a62e52-84a9-4406-b26d-6bdf195697e3\" (UID: \"51a62e52-84a9-4406-b26d-6bdf195697e3\") "
Nov 03 22:16:46.487926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.487892    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume" pod "51a62e52-84a9-4406-b26d-6bdf195697e3" (UID: "51a62e52-84a9-4406-b26d-6bdf195697e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:16:46.490515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.490490    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch" pod "51a62e52-84a9-4406-b26d-6bdf195697e3" (UID: "51a62e52-84a9-4406-b26d-6bdf195697e3"). InnerVolumeSpecName "kube-api-access-4k8ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:46.588718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.588655    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\") pod \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") "
Nov 03 22:16:46.588718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.588701    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\" (UniqueName: \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\") pod \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\" (UID: \"5b1e6bba-caa2-4bed-a303-b36f74e24a05\") "
Nov 03 22:16:46.588971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.588915    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k8ch\" (UniqueName: \"kubernetes.io/projected/51a62e52-84a9-4406-b26d-6bdf195697e3-kube-api-access-4k8ch\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:46.588971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.588926    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a62e52-84a9-4406-b26d-6bdf195697e3-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:46.589101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.589074    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume" pod "5b1e6bba-caa2-4bed-a303-b36f74e24a05" (UID: "5b1e6bba-caa2-4bed-a303-b36f74e24a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:16:46.590765 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.590743    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs" pod "5b1e6bba-caa2-4bed-a303-b36f74e24a05" (UID: "5b1e6bba-caa2-4bed-a303-b36f74e24a05"). InnerVolumeSpecName "kube-api-access-8tprs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:16:46.689893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.689863    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1e6bba-caa2-4bed-a303-b36f74e24a05-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:46.689893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.689888    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tprs\" (UniqueName: \"kubernetes.io/projected/5b1e6bba-caa2-4bed-a303-b36f74e24a05-kube-api-access-8tprs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:16:46.806732 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.806698    5533 generic.go:356] "Generic (PLEG): container finished" podID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" containerID="c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732" exitCode=0
Nov 03 22:16:46.806876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.806744    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"
Nov 03 22:16:46.806876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.806785    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" event={"ID":"5b1e6bba-caa2-4bed-a303-b36f74e24a05","Type":"ContainerDied","Data":"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"}
Nov 03 22:16:46.806876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.806834    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc" event={"ID":"5b1e6bba-caa2-4bed-a303-b36f74e24a05","Type":"ContainerDied","Data":"aabddbc73bdf063a54e4f5f6069ba2b06eb53e5de343d9f35651ebb77e365108"}
Nov 03 22:16:46.806876 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.806856    5533 scope.go:122] "RemoveContainer" containerID="c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"
Nov 03 22:16:46.808297 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.808275    5533 generic.go:356] "Generic (PLEG): container finished" podID="51a62e52-84a9-4406-b26d-6bdf195697e3" containerID="9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e" exitCode=0
Nov 03 22:16:46.808407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.808307    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"
Nov 03 22:16:46.808407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.808364    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" event={"ID":"51a62e52-84a9-4406-b26d-6bdf195697e3","Type":"ContainerDied","Data":"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"}
Nov 03 22:16:46.808491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.808406    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg" event={"ID":"51a62e52-84a9-4406-b26d-6bdf195697e3","Type":"ContainerDied","Data":"bb89ea954a8e001bbf92c034b66601f5d62b139f6cb96f5bff7f6349c538633b"}
Nov 03 22:16:46.816322 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.816305    5533 scope.go:122] "RemoveContainer" containerID="c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"
Nov 03 22:16:46.816552 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:16:46.816536    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732\": container with ID starting with c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732 not found: ID does not exist" containerID="c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"
Nov 03 22:16:46.816605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.816560    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732"} err="failed to get container status \"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732\": rpc error: code = NotFound desc = could not find container \"c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732\": container with ID starting with c854f53fed679075d189a68388c9c50a8c3ce72781f220b9bfe0e6d745bfc732 not found: ID does not exist"
Nov 03 22:16:46.816605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.816576    5533 scope.go:122] "RemoveContainer" containerID="9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"
Nov 03 22:16:46.824348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.824329    5533 scope.go:122] "RemoveContainer" containerID="9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"
Nov 03 22:16:46.824600 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:16:46.824581    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e\": container with ID starting with 9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e not found: ID does not exist" containerID="9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"
Nov 03 22:16:46.824682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.824606    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e"} err="failed to get container status \"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e\": rpc error: code = NotFound desc = could not find container \"9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e\": container with ID starting with 9e20f545af78c10c7e2e45127a0a1c16604f0831315f92cd15ac4d6f5a38b84e not found: ID does not exist"
Nov 03 22:16:46.962452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.962428    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:16:46.967533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:46.967508    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60736a-595fdkzksc"]
Nov 03 22:16:47.071517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:47.071484    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:16:47.074677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:47.074653    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60736a-7dd6457fbw4pbg"]
Nov 03 22:16:48.491350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:48.491316    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a62e52-84a9-4406-b26d-6bdf195697e3" path="/var/lib/kubelet/pods/51a62e52-84a9-4406-b26d-6bdf195697e3/volumes"
Nov 03 22:16:48.491770 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:16:48.491683    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" path="/var/lib/kubelet/pods/5b1e6bba-caa2-4bed-a303-b36f74e24a05/volumes"
Nov 03 22:16:49.260598 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:16:49.260551    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:17:19.259421 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:17:19.259388    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:17:22.518477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:22.518444    5533 scope.go:122] "RemoveContainer" containerID="0901e1e63dc678c9e438ce4bab9fd3a57d470bd0e7f3c403a5f6294988345f22"
Nov 03 22:17:26.045995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:26.045957    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"]
Nov 03 22:17:26.092771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:26.092736    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-zxf69"]
Nov 03 22:17:26.491900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:26.491865    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c50f75b-8632-406b-8796-d4776752cf53" path="/var/lib/kubelet/pods/6c50f75b-8632-406b-8796-d4776752cf53/volumes"
Nov 03 22:17:27.090331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:27.090296    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:17:29.362811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.362779    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"]
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363133    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" containerName="payload-pre-processing"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363145    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" containerName="payload-pre-processing"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363166    5533 cpu_manager.go:414] "containerMap: removing container" podUID="51a62e52-84a9-4406-b26d-6bdf195697e3" containerName="payload-processing"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363172    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a62e52-84a9-4406-b26d-6bdf195697e3" containerName="payload-processing"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363189    5533 cpu_manager.go:414] "containerMap: removing container" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363194    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363204    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1decd0f1-6b66-4777-984a-72662e3f2d99" containerName="revoke-keys"
Nov 03 22:17:29.363221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363209    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1decd0f1-6b66-4777-984a-72662e3f2d99" containerName="revoke-keys"
Nov 03 22:17:29.363491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363259    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1e6bba-caa2-4bed-a303-b36f74e24a05" containerName="payload-pre-processing"
Nov 03 22:17:29.363491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363271    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a62e52-84a9-4406-b26d-6bdf195697e3" containerName="payload-processing"
Nov 03 22:17:29.363491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363277    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacc5cb5-3b9b-48f9-8113-421c0bdb9323" containerName="cleanup"
Nov 03 22:17:29.363491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.363285    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1decd0f1-6b66-4777-984a-72662e3f2d99" containerName="revoke-keys"
Nov 03 22:17:29.366084 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.366065    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.415030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.415006    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:17:29.448834 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.448803    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.448982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.448852    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88d8m\" (UniqueName: \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.449074 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.449047    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.461711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.461688    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:17:29.464172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.464151    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"]
Nov 03 22:17:29.550534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.550501    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.550649 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.550569    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.550649 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.550617    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88d8m\" (UniqueName: \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.551232 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.551211    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.552862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.552844    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.608461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.608437    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88d8m\" (UniqueName: \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:29.677458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:29.677431    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:30.043615 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:30.043587    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"]
Nov 03 22:17:31.061374 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:31.061347    5533 generic.go:356] "Generic (PLEG): container finished" podID="7bde13f6-c55a-45a5-9dd0-a3e722507359" containerID="beff191f2afd23a0b45f934a04e8cf9b7344e8e386a5a4919f68cbc19ce4d090" exitCode=0
Nov 03 22:17:31.061714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:31.061389    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd" event={"ID":"7bde13f6-c55a-45a5-9dd0-a3e722507359","Type":"ContainerDied","Data":"beff191f2afd23a0b45f934a04e8cf9b7344e8e386a5a4919f68cbc19ce4d090"}
Nov 03 22:17:31.061714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:31.061433    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd" event={"ID":"7bde13f6-c55a-45a5-9dd0-a3e722507359","Type":"ContainerStarted","Data":"7f5449c805907450ea75d1243bb8ec699e2f7bda6abd53a8610d1942086b2aa3"}
Nov 03 22:17:32.204389 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.204364    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:32.275349 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.275322    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\") pod \"7bde13f6-c55a-45a5-9dd0-a3e722507359\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") "
Nov 03 22:17:32.275488 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.275443    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\") pod \"7bde13f6-c55a-45a5-9dd0-a3e722507359\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") "
Nov 03 22:17:32.275536 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.275522    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\" (UniqueName: \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\") pod \"7bde13f6-c55a-45a5-9dd0-a3e722507359\" (UID: \"7bde13f6-c55a-45a5-9dd0-a3e722507359\") "
Nov 03 22:17:32.275888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.275864    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca" pod "7bde13f6-c55a-45a5-9dd0-a3e722507359" (UID: "7bde13f6-c55a-45a5-9dd0-a3e722507359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:17:32.277499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.277472    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca" pod "7bde13f6-c55a-45a5-9dd0-a3e722507359" (UID: "7bde13f6-c55a-45a5-9dd0-a3e722507359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:17:32.277596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.277502    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m" pod "7bde13f6-c55a-45a5-9dd0-a3e722507359" (UID: "7bde13f6-c55a-45a5-9dd0-a3e722507359"). InnerVolumeSpecName "kube-api-access-88d8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:32.376491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.376428    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88d8m\" (UniqueName: \"kubernetes.io/projected/7bde13f6-c55a-45a5-9dd0-a3e722507359-kube-api-access-88d8m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:32.376491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.376455    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:32.376491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:32.376466    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bde13f6-c55a-45a5-9dd0-a3e722507359-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:33.073279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:33.073249    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"
Nov 03 22:17:33.073465 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:33.073249    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd" event={"ID":"7bde13f6-c55a-45a5-9dd0-a3e722507359","Type":"ContainerDied","Data":"7f5449c805907450ea75d1243bb8ec699e2f7bda6abd53a8610d1942086b2aa3"}
Nov 03 22:17:33.073465 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:33.073372    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5449c805907450ea75d1243bb8ec699e2f7bda6abd53a8610d1942086b2aa3"
Nov 03 22:17:49.259848 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:17:49.259813    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:17:49.874154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:49.874113    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"]
Nov 03 22:17:49.920492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:49.920463    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60736a-nllrd"]
Nov 03 22:17:50.118861 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:50.118827    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:17:50.119374 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:50.119320    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy" containerID="cri-o://4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30" gracePeriod=30
Nov 03 22:17:50.492361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:50.492327    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bde13f6-c55a-45a5-9dd0-a3e722507359" path="/var/lib/kubelet/pods/7bde13f6-c55a-45a5-9dd0-a3e722507359/volumes"
Nov 03 22:17:54.063454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:54.063416    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:17:55.371484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.371459    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:17:55.476419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476393    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476450    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476480    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476517    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476577    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476582    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476606    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476665    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476712    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.476786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476769    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\") pod \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\" (UID: \"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3\") "
Nov 03 22:17:55.477186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476796    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:55.477186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476831    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:55.477186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.476920    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:55.477369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.477214    5533 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-data\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.477369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.477237    5533 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.477369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.477237    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:17:55.477369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.477250    5533 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-workload-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.477369 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.477291    5533 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-credential-socket\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.478892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.478870    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:55.479164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.479137    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "kube-api-access-2w86r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:55.479402 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.479378    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:17:55.479510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.479468    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token" pod "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" (UID: "e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:55.578058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.578031    5533 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-envoy\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.578058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.578056    5533 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istiod-ca-cert\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.578230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.578068    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w86r\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-kube-api-access-2w86r\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.578230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.578077    5533 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-podinfo\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:55.578230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:55.578086    5533 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3-istio-token\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:17:56.197481 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.197445    5533 generic.go:356] "Generic (PLEG): container finished" podID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerID="4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30" exitCode=0
Nov 03 22:17:56.197693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.197491    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"
Nov 03 22:17:56.197693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.197527    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerDied","Data":"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"}
Nov 03 22:17:56.197693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.197588    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm" event={"ID":"e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3","Type":"ContainerDied","Data":"66501cbc8648c84a0e616273711b41be8ab047f5870edfebdb056d7738a6ac36"}
Nov 03 22:17:56.197693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.197613    5533 scope.go:122] "RemoveContainer" containerID="4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"
Nov 03 22:17:56.210468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.210448    5533 scope.go:122] "RemoveContainer" containerID="31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"
Nov 03 22:17:56.221073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.221054    5533 scope.go:122] "RemoveContainer" containerID="4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"
Nov 03 22:17:56.221395 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:17:56.221372    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30\": container with ID starting with 4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30 not found: ID does not exist" containerID="4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"
Nov 03 22:17:56.221505 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.221408    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30"} err="failed to get container status \"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30\": rpc error: code = NotFound desc = could not find container \"4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30\": container with ID starting with 4552fd96112168985d3943aafe67f118f231f238ee6f9f4df5331133d1edca30 not found: ID does not exist"
Nov 03 22:17:56.221505 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.221432    5533 scope.go:122] "RemoveContainer" containerID="31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"
Nov 03 22:17:56.221714 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:17:56.221694    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417\": container with ID starting with 31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417 not found: ID does not exist" containerID="31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"
Nov 03 22:17:56.221772 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.221720    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417"} err="failed to get container status \"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417\": rpc error: code = NotFound desc = could not find container \"31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417\": container with ID starting with 31a2b641f100d47214f24cd77d3698172824db06f37966901cec985e761f4417 not found: ID does not exist"
Nov 03 22:17:56.400466 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.400440    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:17:56.446954 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.446929    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60736a-openshift-default-6f5df7b66f89fgm"]
Nov 03 22:17:56.491726 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:56.491673    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" path="/var/lib/kubelet/pods/e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3/volumes"
Nov 03 22:17:57.095171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095138    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:17:57.095752 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095733    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:17:57.095805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095757    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:17:57.095805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095778    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7bde13f6-c55a-45a5-9dd0-a3e722507359" containerName="revoke-keys"
Nov 03 22:17:57.095805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095786    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bde13f6-c55a-45a5-9dd0-a3e722507359" containerName="revoke-keys"
Nov 03 22:17:57.095913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095879    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bde13f6-c55a-45a5-9dd0-a3e722507359" containerName="revoke-keys"
Nov 03 22:17:57.095913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.095894    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:17:57.101003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.100980    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.151590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.151564    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bb094892-kserve-self-signed-certs\""
Nov 03 22:17:57.192882 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.192855    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcl6x\" (UniqueName: \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.193015 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.192897    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.193015 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.192923    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.193114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.193091    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.193153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.193126    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.193210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.193152    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.200844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.200821    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:17:57.294104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294069    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcl6x\" (UniqueName: \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294123    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294151    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294223    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294241    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294262    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294617    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294770 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294666    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.294887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.294868    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.296621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.296597    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.296849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.296832    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.351548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.351492    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcl6x\" (UniqueName: \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\") pod \"e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") " pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.417730 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.417702    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:57.790992 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:57.790966    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:17:57.792914 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:17:57.792880    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a9c6ec_5ebf_44ac_9a86_a0f67f0034d6.slice/crio-e1a29b980af0b8f2e4648eaeacd5e8323ee203ced9c49bd1f956eba884c0f0d0 WatchSource:0}: Error finding container e1a29b980af0b8f2e4648eaeacd5e8323ee203ced9c49bd1f956eba884c0f0d0: Status 404 returned error can't find the container with id e1a29b980af0b8f2e4648eaeacd5e8323ee203ced9c49bd1f956eba884c0f0d0
Nov 03 22:17:58.212937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:58.212904    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" event={"ID":"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6","Type":"ContainerStarted","Data":"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"}
Nov 03 22:17:58.212937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:58.212943    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" event={"ID":"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6","Type":"ContainerStarted","Data":"e1a29b980af0b8f2e4648eaeacd5e8323ee203ced9c49bd1f956eba884c0f0d0"}
Nov 03 22:17:58.324444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:58.324395    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" podStartSLOduration=1.324381732 podStartE2EDuration="1.324381732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:58.322855697 +0000 UTC m=+5379.954169193" watchObservedRunningTime="2026-11-03 22:17:58.324381732 +0000 UTC m=+5379.955695209"
Nov 03 22:17:58.418219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:58.418191    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:17:59.045133 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:59.045099    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"]
Nov 03 22:17:59.091739 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:17:59.091711    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-t78fh"]
Nov 03 22:18:00.492106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:00.492071    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7273cdc7-2f31-403d-b4c8-3027674f87b4" path="/var/lib/kubelet/pods/7273cdc7-2f31-403d-b4c8-3027674f87b4/volumes"
Nov 03 22:18:07.106034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:07.105996    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:18:07.418424 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:07.418346    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:18:07.430472 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:07.430448    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:18:08.286069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:08.286038    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:18:19.046404 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.046369    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"]
Nov 03 22:18:19.092899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.092871    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fkwvh"]
Nov 03 22:18:19.260845 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:18:19.260808    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:18:19.483221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.483200    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:18:19.494506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.494486    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:18:19.535522 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.535504    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:18:19.546093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:19.546074    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:18:20.491319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:20.491286    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5226da-d475-41ae-9e6d-94c4b3bc435c" path="/var/lib/kubelet/pods/be5226da-d475-41ae-9e6d-94c4b3bc435c/volumes"
Nov 03 22:18:22.546492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:22.546459    5533 scope.go:122] "RemoveContainer" containerID="8eb93d50a448173d97381cf764282ca1d32e0584f4d4064818731a7b46d29509"
Nov 03 22:18:22.555024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:22.555004    5533 scope.go:122] "RemoveContainer" containerID="97b78c2f32e4dd08e00561bc433692654dbec521d162eb0607dffe750cb912c4"
Nov 03 22:18:24.640199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.640165    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:18:24.640581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.640515    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:18:24.640581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.640525    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:18:24.640685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.640601    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a1ce01-c0d2-43c6-a45c-4f8f0a57d7c3" containerName="istio-proxy"
Nov 03 22:18:24.644957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.644937    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725378 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725348    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725508    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725526    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725558    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.725688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.725610    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pg2\" (UniqueName: \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.743367 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.743342    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bb094892-kserve-self-signed-certs\""
Nov 03 22:18:24.745350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.745331    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:18:24.826479 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826443    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.826692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826498    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.826692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826515    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.826692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826545    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.826692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826562    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pg2\" (UniqueName: \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.826692 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.826592    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.827090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.827066    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.827173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.827115    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.827173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.827148    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.828812 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.828791    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.829089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.829070    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.884035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.884006    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pg2\" (UniqueName: \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\") pod \"e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") " pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:24.956810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:24.956785    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:25.332139 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:25.332111    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:18:25.334115 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:18:25.334083    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53b1dc1_ab7f_4ced_8a69_7eaea1e92115.slice/crio-d9b0bb196f44d806d3c2432eba4714e4a5b9ce2a633a2ac8d0718127205843dc WatchSource:0}: Error finding container d9b0bb196f44d806d3c2432eba4714e4a5b9ce2a633a2ac8d0718127205843dc: Status 404 returned error can't find the container with id d9b0bb196f44d806d3c2432eba4714e4a5b9ce2a633a2ac8d0718127205843dc
Nov 03 22:18:25.369613 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:25.369583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" event={"ID":"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115","Type":"ContainerStarted","Data":"d9b0bb196f44d806d3c2432eba4714e4a5b9ce2a633a2ac8d0718127205843dc"}
Nov 03 22:18:26.376266 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:26.376232    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" event={"ID":"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115","Type":"ContainerStarted","Data":"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"}
Nov 03 22:18:26.485908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:26.485842    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" podStartSLOduration=2.485826813 podStartE2EDuration="2.485826813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:26.485678852 +0000 UTC m=+5408.116992345" watchObservedRunningTime="2026-11-03 22:18:26.485826813 +0000 UTC m=+5408.117140286"
Nov 03 22:18:26.957164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:26.957130    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:29.492583 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:29.492501    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:18:31.050263 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:31.050227    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"]
Nov 03 22:18:31.095721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:31.095690    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vqjb9"]
Nov 03 22:18:32.517655 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:32.517595    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b274855d-f6eb-4bff-9c00-8df0b1c37a68" path="/var/lib/kubelet/pods/b274855d-f6eb-4bff-9c00-8df0b1c37a68/volumes"
Nov 03 22:18:34.957869 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:34.957836    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:34.970678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:34.970651    5533 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:35.457431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:35.457405    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:18:49.259328 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:18:49.259288    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:18:58.540879 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.540842    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"]
Nov 03 22:18:58.544204 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.544178    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.639781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.639750    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:18:58.639922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.639790    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:18:58.641920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.641900    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"]
Nov 03 22:18:58.643740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.643712    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.643804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.643774    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.643893 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.643879    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55bb\" (UniqueName: \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.744920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.744894    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55bb\" (UniqueName: \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.745051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.744934    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.745051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.744983    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.745622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.745595    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.747337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.747319    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.802541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.802480    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55bb\" (UniqueName: \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\") pod \"maas-api-revoke-sub-452b6d7b-5x9p2\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:58.857112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:58.857082    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:18:59.219605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:59.219580    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"]
Nov 03 22:18:59.221687 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:18:59.221657    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523a5f01_e51d_470d_99cf_6fa730d61d30.slice/crio-f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b WatchSource:0}: Error finding container f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b: Status 404 returned error can't find the container with id f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b
Nov 03 22:18:59.569180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:59.569146    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2" event={"ID":"523a5f01-e51d-470d-99cf-6fa730d61d30","Type":"ContainerStarted","Data":"275fdcac126239f16dd3cad2e7fc9a7c747dabe5af965827543baf5eac648b5c"}
Nov 03 22:18:59.569554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:59.569200    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2" event={"ID":"523a5f01-e51d-470d-99cf-6fa730d61d30","Type":"ContainerStarted","Data":"f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b"}
Nov 03 22:18:59.679856 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:18:59.679803    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2" podStartSLOduration=1.6797857999999999 podStartE2EDuration="1.6797858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:59.677964944 +0000 UTC m=+5441.309278423" watchObservedRunningTime="2026-11-03 22:18:59.6797858 +0000 UTC m=+5441.311099281"
Nov 03 22:19:00.004796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:00.004763    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:19:00.578880 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:00.578841    5533 generic.go:356] "Generic (PLEG): container finished" podID="523a5f01-e51d-470d-99cf-6fa730d61d30" containerID="275fdcac126239f16dd3cad2e7fc9a7c747dabe5af965827543baf5eac648b5c" exitCode=0
Nov 03 22:19:00.579393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:00.578969    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2" event={"ID":"523a5f01-e51d-470d-99cf-6fa730d61d30","Type":"ContainerDied","Data":"275fdcac126239f16dd3cad2e7fc9a7c747dabe5af965827543baf5eac648b5c"}
Nov 03 22:19:02.024541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.024500    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:19:02.085954 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.085922    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\") pod \"523a5f01-e51d-470d-99cf-6fa730d61d30\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") "
Nov 03 22:19:02.086208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.086180    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\") pod \"523a5f01-e51d-470d-99cf-6fa730d61d30\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") "
Nov 03 22:19:02.086424 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.086405    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\" (UniqueName: \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\") pod \"523a5f01-e51d-470d-99cf-6fa730d61d30\" (UID: \"523a5f01-e51d-470d-99cf-6fa730d61d30\") "
Nov 03 22:19:02.087461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.087436    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca" pod "523a5f01-e51d-470d-99cf-6fa730d61d30" (UID: "523a5f01-e51d-470d-99cf-6fa730d61d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:02.109109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.109039    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca" pod "523a5f01-e51d-470d-99cf-6fa730d61d30" (UID: "523a5f01-e51d-470d-99cf-6fa730d61d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:02.121495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.121420    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb" pod "523a5f01-e51d-470d-99cf-6fa730d61d30" (UID: "523a5f01-e51d-470d-99cf-6fa730d61d30"). InnerVolumeSpecName "kube-api-access-n55bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:02.200706 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.195188    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55bb\" (UniqueName: \"kubernetes.io/projected/523a5f01-e51d-470d-99cf-6fa730d61d30-kube-api-access-n55bb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:02.200706 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.195222    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:02.200706 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.195241    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523a5f01-e51d-470d-99cf-6fa730d61d30-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:02.620656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.619723    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2" event={"ID":"523a5f01-e51d-470d-99cf-6fa730d61d30","Type":"ContainerDied","Data":"f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b"}
Nov 03 22:19:02.620656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.619807    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0dc4d217bd1fa975f38c5f1c3b5d5a092e64547241b6d3fb9ca63cecb2be45b"
Nov 03 22:19:02.620656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:02.619829    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"
Nov 03 22:19:11.447671 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:11.447623    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:19:15.586481 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:15.586446    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:19:19.259835 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:19:19.259797    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:19:22.583443 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:22.583409    5533 scope.go:122] "RemoveContainer" containerID="b3473f77e8678869a736239e832ead8a6574323e3a68b59ffe9fa80606eb625e"
Nov 03 22:19:22.592250 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:22.592229    5533 scope.go:122] "RemoveContainer" containerID="5d5539b0dade39fb564d8d0b508451b1e8b50a3a0608841c20b3da264985118f"
Nov 03 22:19:26.248785 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.248753    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:19:26.467157 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.467120    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" podUID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" containerName="main" containerID="cri-o://1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7" gracePeriod=45
Nov 03 22:19:26.712440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.712416    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:19:26.773518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.773486    5533 generic.go:356] "Generic (PLEG): container finished" podID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" containerID="1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7" exitCode=0
Nov 03 22:19:26.773621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.773518    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"
Nov 03 22:19:26.773621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.773573    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" event={"ID":"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6","Type":"ContainerDied","Data":"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"}
Nov 03 22:19:26.773792 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.773623    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf" event={"ID":"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6","Type":"ContainerDied","Data":"e1a29b980af0b8f2e4648eaeacd5e8323ee203ced9c49bd1f956eba884c0f0d0"}
Nov 03 22:19:26.773792 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.773673    5533 scope.go:122] "RemoveContainer" containerID="1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"
Nov 03 22:19:26.782124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.782108    5533 scope.go:122] "RemoveContainer" containerID="1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"
Nov 03 22:19:26.782383 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:19:26.782364    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7\": container with ID starting with 1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7 not found: ID does not exist" containerID="1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"
Nov 03 22:19:26.782455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.782395    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7"} err="failed to get container status \"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7\": rpc error: code = NotFound desc = could not find container \"1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7\": container with ID starting with 1a99d309a0e37c308082299b3936a6545741a4534959ed2aea6419a6784e97c7 not found: ID does not exist"
Nov 03 22:19:26.805130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805108    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805169    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\" (UniqueName: \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805221 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805210    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805305 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805245    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805305 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805264    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805310    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\") pod \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\" (UID: \"56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6\") "
Nov 03 22:19:26.805393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805358    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:26.805549 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805502    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:26.805689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805559    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:26.805689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805661    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:26.805689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805674    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:26.805689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.805683    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:26.807226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.807204    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "kube-api-access-bcl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:26.807315 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.807300    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:26.807433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.807412    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs" pod "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" (UID: "56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:26.906254 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.906205    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:26.906254 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.906226    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcl6x\" (UniqueName: \"kubernetes.io/projected/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-kube-api-access-bcl6x\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:26.906254 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:26.906235    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:27.203227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:27.203197    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:19:27.249578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:27.249552    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bb094892-kserve-cb6578685-2xvvf"]
Nov 03 22:19:28.492734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:28.492701    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" path="/var/lib/kubelet/pods/56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6/volumes"
Nov 03 22:19:30.540619 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.540579    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"]
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.540977    5533 cpu_manager.go:414] "containerMap: removing container" podUID="523a5f01-e51d-470d-99cf-6fa730d61d30" containerName="revoke-keys"
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.540991    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523a5f01-e51d-470d-99cf-6fa730d61d30" containerName="revoke-keys"
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.541018    5533 cpu_manager.go:414] "containerMap: removing container" podUID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" containerName="main"
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.541023    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" containerName="main"
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.541081    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="523a5f01-e51d-470d-99cf-6fa730d61d30" containerName="revoke-keys"
Nov 03 22:19:30.541110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.541093    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a9c6ec-5ebf-44ac-9a86-a0f67f0034d6" containerName="main"
Nov 03 22:19:30.545353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.545332    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.598699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.598667    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"]
Nov 03 22:19:30.639741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.639714    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.639884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.639774    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vr5\" (UniqueName: \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.639936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.639894    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.642838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.642815    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:19:30.642953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.642876    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:19:30.740951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.740925    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.741074 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.740968    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.741074 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.741018    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vr5\" (UniqueName: \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.741611 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.741587    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.743239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.743216    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.763191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.763168    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:19:30.803024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.802974    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vr5\" (UniqueName: \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\") pod \"maas-api-revoke-sub-3e31a506-2gxw9\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:30.861827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:30.861805    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:31.230800 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:31.230766    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"]
Nov 03 22:19:31.804407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:31.804308    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9" event={"ID":"6cc556fc-7177-4d40-9ae7-557c84257a3a","Type":"ContainerStarted","Data":"39990a72d4857dc7159171a03ca2e8784c356482bc4ad6aa5580740ab581b6fe"}
Nov 03 22:19:31.804407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:31.804371    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9" event={"ID":"6cc556fc-7177-4d40-9ae7-557c84257a3a","Type":"ContainerStarted","Data":"35349cc7599f7f098676f4057b0d5d10c2e6f7978983aa7a801cdf5a24570f91"}
Nov 03 22:19:31.914604 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:31.914498    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9" podStartSLOduration=1.9144782710000001 podStartE2EDuration="1.914478271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:31.912902916 +0000 UTC m=+5473.544216396" watchObservedRunningTime="2026-11-03 22:19:31.914478271 +0000 UTC m=+5473.545791749"
Nov 03 22:19:32.827667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:32.827405    5533 generic.go:356] "Generic (PLEG): container finished" podID="6cc556fc-7177-4d40-9ae7-557c84257a3a" containerID="39990a72d4857dc7159171a03ca2e8784c356482bc4ad6aa5580740ab581b6fe" exitCode=0
Nov 03 22:19:32.827667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:32.827485    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9" event={"ID":"6cc556fc-7177-4d40-9ae7-557c84257a3a","Type":"ContainerDied","Data":"39990a72d4857dc7159171a03ca2e8784c356482bc4ad6aa5580740ab581b6fe"}
Nov 03 22:19:33.980580 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:33.980555    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:34.126925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.126859    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\") pod \"6cc556fc-7177-4d40-9ae7-557c84257a3a\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") "
Nov 03 22:19:34.127075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.127028    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\") pod \"6cc556fc-7177-4d40-9ae7-557c84257a3a\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") "
Nov 03 22:19:34.127075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.127051    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\" (UniqueName: \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\") pod \"6cc556fc-7177-4d40-9ae7-557c84257a3a\" (UID: \"6cc556fc-7177-4d40-9ae7-557c84257a3a\") "
Nov 03 22:19:34.127386 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.127360    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca" pod "6cc556fc-7177-4d40-9ae7-557c84257a3a" (UID: "6cc556fc-7177-4d40-9ae7-557c84257a3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:34.128953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.128930    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca" pod "6cc556fc-7177-4d40-9ae7-557c84257a3a" (UID: "6cc556fc-7177-4d40-9ae7-557c84257a3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:34.129113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.129099    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5" pod "6cc556fc-7177-4d40-9ae7-557c84257a3a" (UID: "6cc556fc-7177-4d40-9ae7-557c84257a3a"). InnerVolumeSpecName "kube-api-access-c9vr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:34.228687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.228664    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:34.228687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.228687    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vr5\" (UniqueName: \"kubernetes.io/projected/6cc556fc-7177-4d40-9ae7-557c84257a3a-kube-api-access-c9vr5\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:34.228816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.228697    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc556fc-7177-4d40-9ae7-557c84257a3a-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:34.841206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.841128    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"
Nov 03 22:19:34.841206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.841153    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9" event={"ID":"6cc556fc-7177-4d40-9ae7-557c84257a3a","Type":"ContainerDied","Data":"35349cc7599f7f098676f4057b0d5d10c2e6f7978983aa7a801cdf5a24570f91"}
Nov 03 22:19:34.841206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:34.841203    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35349cc7599f7f098676f4057b0d5d10c2e6f7978983aa7a801cdf5a24570f91"
Nov 03 22:19:41.268184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.268131    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerName="main" containerID="cri-o://371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887" gracePeriod=45
Nov 03 22:19:41.529543 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.529480    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:19:41.589652 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589609    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\" (UniqueName: \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.589807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589685    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.589807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589717    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.589807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589782    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.589941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589809    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\" (UniqueName: \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.589941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589832    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\") pod \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\" (UID: \"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115\") "
Nov 03 22:19:41.590060 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589969    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.590060 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.589997    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.590182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.590070    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.590245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.590178    5533 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-home\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.590245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.590198    5533 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tmp-dir\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.590245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.590214    5533 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-model-cache\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.591849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.591826    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:41.591966 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.591874    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "kube-api-access-r6pg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:41.591966 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.591958    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm" pod "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" (UID: "d53b1dc1-ab7f-4ced-8a69-7eaea1e92115"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:41.691382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.691356    5533 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-tls-certs\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.691382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.691381    5533 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-dshm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.691562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.691392    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pg2\" (UniqueName: \"kubernetes.io/projected/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115-kube-api-access-r6pg2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.882552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.882491    5533 generic.go:356] "Generic (PLEG): container finished" podID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerID="371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887" exitCode=0
Nov 03 22:19:41.882552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.882536    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"
Nov 03 22:19:41.882552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.882546    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" event={"ID":"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115","Type":"ContainerDied","Data":"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"}
Nov 03 22:19:41.882858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.882578    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" event={"ID":"d53b1dc1-ab7f-4ced-8a69-7eaea1e92115","Type":"ContainerDied","Data":"d9b0bb196f44d806d3c2432eba4714e4a5b9ce2a633a2ac8d0718127205843dc"}
Nov 03 22:19:41.882858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.882600    5533 scope.go:122] "RemoveContainer" containerID="371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"
Nov 03 22:19:41.891973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.891957    5533 scope.go:122] "RemoveContainer" containerID="371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"
Nov 03 22:19:41.892226 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:19:41.892210    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887\": container with ID starting with 371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887 not found: ID does not exist" containerID="371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"
Nov 03 22:19:41.892274 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:41.892235    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887"} err="failed to get container status \"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887\": rpc error: code = NotFound desc = could not find container \"371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887\": container with ID starting with 371f882ddea0d3396115b288daf813a3ec4c999647c384455e945dd62824e887 not found: ID does not exist"
Nov 03 22:19:42.039677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.039650    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:19:42.043163 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.043131    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws"]
Nov 03 22:19:42.085941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.085915    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"]
Nov 03 22:19:42.087557 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.087535    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111335a5-nsz4f"]
Nov 03 22:19:42.446444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.446393    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-bb094892-kserve-8648495fcc-rxfws" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerName="main" probeResult="failure" output="Get \"https://10.133.1.113:8000/ready\": context deadline exceeded"
Nov 03 22:19:42.491799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.491773    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc86db23-88e6-434d-bea7-90270aaed199" path="/var/lib/kubelet/pods/bc86db23-88e6-434d-bea7-90270aaed199/volumes"
Nov 03 22:19:42.492178 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:42.492165    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" path="/var/lib/kubelet/pods/d53b1dc1-ab7f-4ced-8a69-7eaea1e92115/volumes"
Nov 03 22:19:43.272910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:43.272876    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:19:49.261116 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:19:49.261067    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:19:54.696473 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:54.696435    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:19:56.299759 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.299720    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"]
Nov 03 22:19:56.300171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300113    5533 cpu_manager.go:414] "containerMap: removing container" podUID="6cc556fc-7177-4d40-9ae7-557c84257a3a" containerName="revoke-keys"
Nov 03 22:19:56.300171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300125    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc556fc-7177-4d40-9ae7-557c84257a3a" containerName="revoke-keys"
Nov 03 22:19:56.300171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300140    5533 cpu_manager.go:414] "containerMap: removing container" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerName="main"
Nov 03 22:19:56.300171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300145    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerName="main"
Nov 03 22:19:56.300317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300202    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc556fc-7177-4d40-9ae7-557c84257a3a" containerName="revoke-keys"
Nov 03 22:19:56.300317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.300216    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53b1dc1-ab7f-4ced-8a69-7eaea1e92115" containerName="main"
Nov 03 22:19:56.304768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.304751    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.355448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.355424    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:19:56.386939 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.386910    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.387071 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.386969    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxj2r\" (UniqueName: \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.387071 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.387046    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.402154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.402129    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:19:56.404116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.404096    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"]
Nov 03 22:19:56.487619 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.487593    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.487793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.487664    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxj2r\" (UniqueName: \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.487793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.487697    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.488259 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.488238    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.490103 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.490083    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.545108 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.545086    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxj2r\" (UniqueName: \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\") pod \"maas-api-revoke-sub-8feec754-wxstt\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.618174 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.618110    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:56.981116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:56.981087    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"]
Nov 03 22:19:56.983022 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:19:56.982984    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7122e4b5_267b_49a7_aeb7_5302dc1884e9.slice/crio-05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e WatchSource:0}: Error finding container 05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e: Status 404 returned error can't find the container with id 05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e
Nov 03 22:19:57.007688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:57.007659    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt" event={"ID":"7122e4b5-267b-49a7-aeb7-5302dc1884e9","Type":"ContainerStarted","Data":"05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e"}
Nov 03 22:19:58.021056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:58.020981    5533 generic.go:356] "Generic (PLEG): container finished" podID="7122e4b5-267b-49a7-aeb7-5302dc1884e9" containerID="50dc9975ca791be3028513fbd1972a22c93e3ce5df21a456b93c411d98ec9866" exitCode=0
Nov 03 22:19:58.021056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:58.021046    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt" event={"ID":"7122e4b5-267b-49a7-aeb7-5302dc1884e9","Type":"ContainerDied","Data":"50dc9975ca791be3028513fbd1972a22c93e3ce5df21a456b93c411d98ec9866"}
Nov 03 22:19:59.152520 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.152496    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:19:59.208734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.208711    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\") pod \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") "
Nov 03 22:19:59.208866 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.208797    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\" (UniqueName: \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\") pod \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") "
Nov 03 22:19:59.208973 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.208954    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\") pod \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\" (UID: \"7122e4b5-267b-49a7-aeb7-5302dc1884e9\") "
Nov 03 22:19:59.209129 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.209106    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca" pod "7122e4b5-267b-49a7-aeb7-5302dc1884e9" (UID: "7122e4b5-267b-49a7-aeb7-5302dc1884e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:59.209281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.209261    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:59.210897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.210872    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r" pod "7122e4b5-267b-49a7-aeb7-5302dc1884e9" (UID: "7122e4b5-267b-49a7-aeb7-5302dc1884e9"). InnerVolumeSpecName "kube-api-access-hxj2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:59.210897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.210881    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca" pod "7122e4b5-267b-49a7-aeb7-5302dc1884e9" (UID: "7122e4b5-267b-49a7-aeb7-5302dc1884e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:59.310432 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.310367    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxj2r\" (UniqueName: \"kubernetes.io/projected/7122e4b5-267b-49a7-aeb7-5302dc1884e9-kube-api-access-hxj2r\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:19:59.310432 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:19:59.310397    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7122e4b5-267b-49a7-aeb7-5302dc1884e9-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:20:00.031894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.031861    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"
Nov 03 22:20:00.031894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.031888    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt" event={"ID":"7122e4b5-267b-49a7-aeb7-5302dc1884e9","Type":"ContainerDied","Data":"05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e"}
Nov 03 22:20:00.032187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.031932    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d6732df6865005ea28bb40a9b46cfa9fcf64bcb875462aef6776fe2ce8c32e"
Nov 03 22:20:00.045102 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.045078    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"]
Nov 03 22:20:00.092039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.092016    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75b4649d-9wsr5"]
Nov 03 22:20:00.491473 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:00.491434    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2" path="/var/lib/kubelet/pods/4c4b05fa-852e-4ffc-baaf-7bdfc9a478c2/volumes"
Nov 03 22:20:09.192168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:09.192133    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:20:13.127867 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.127830    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:20:13.128365 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.128343    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7122e4b5-267b-49a7-aeb7-5302dc1884e9" containerName="revoke-keys"
Nov 03 22:20:13.128454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.128368    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7122e4b5-267b-49a7-aeb7-5302dc1884e9" containerName="revoke-keys"
Nov 03 22:20:13.128525 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.128499    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7122e4b5-267b-49a7-aeb7-5302dc1884e9" containerName="revoke-keys"
Nov 03 22:20:13.131533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.131509    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.227290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.227265    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 22:20:13.227464 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.227445    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kbdwc\""
Nov 03 22:20:13.228793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.228775    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86vl\" (UniqueName: \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.228854 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.228804    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.229179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.229160    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:20:13.330290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.330261    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86vl\" (UniqueName: \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.330448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.330296    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.331025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.331006    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.388335 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.388276    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86vl\" (UniqueName: \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\") pod \"payload-pre-processing-c65954f6c-wntm8\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") " pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.444222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.444194    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:13.808687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:13.808652    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:20:13.810250 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:20:13.810209    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef7e1f1_0cb6_40ca_b3c4_382aa770823e.slice/crio-288bb160410efc9c0bc92be2e8055f096cc00b51acf64279d3e8be1d2c86a780 WatchSource:0}: Error finding container 288bb160410efc9c0bc92be2e8055f096cc00b51acf64279d3e8be1d2c86a780: Status 404 returned error can't find the container with id 288bb160410efc9c0bc92be2e8055f096cc00b51acf64279d3e8be1d2c86a780
Nov 03 22:20:14.104783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:14.104705    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" event={"ID":"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e","Type":"ContainerStarted","Data":"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"}
Nov 03 22:20:14.104783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:14.104764    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" event={"ID":"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e","Type":"ContainerStarted","Data":"288bb160410efc9c0bc92be2e8055f096cc00b51acf64279d3e8be1d2c86a780"}
Nov 03 22:20:14.104961 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:14.104794    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:14.213420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:14.213338    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" podStartSLOduration=1.213326607 podStartE2EDuration="1.213326607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:14.213204386 +0000 UTC m=+5515.844517863" watchObservedRunningTime="2026-11-03 22:20:14.213326607 +0000 UTC m=+5515.844640083"
Nov 03 22:20:19.259959 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:20:19.259923    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:20:22.616116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:22.616086    5533 scope.go:122] "RemoveContainer" containerID="faa593e98af66411d5d4ad173b39fcf6f4dff4af3980bbc8da3573ef6a0994e3"
Nov 03 22:20:25.113398 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:25.113365    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:27.310144 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.310111    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:20:27.314988 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.314970    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.369059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.369035    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:20:27.449653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.449594    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5q7\" (UniqueName: \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.449824 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.449657    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.550937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.550899    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5q7\" (UniqueName: \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.551107 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.550945    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.551764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.551741    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.608962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.608912    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5q7\" (UniqueName: \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\") pod \"payload-processing-995bb6bdd-fd56f\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") " pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.627737 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.627712    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:27.997291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:27.997267    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:20:27.998293 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:20:27.998267    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c84915e_e66c_406f_a253_86724e1536cf.slice/crio-32cdb2eefe1830224f2e160e90236d13c2538ce150e1c2368fa8379833f76d58 WatchSource:0}: Error finding container 32cdb2eefe1830224f2e160e90236d13c2538ce150e1c2368fa8379833f76d58: Status 404 returned error can't find the container with id 32cdb2eefe1830224f2e160e90236d13c2538ce150e1c2368fa8379833f76d58
Nov 03 22:20:28.179683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:28.179626    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" event={"ID":"1c84915e-e66c-406f-a253-86724e1536cf","Type":"ContainerStarted","Data":"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"}
Nov 03 22:20:28.179793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:28.179715    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:28.179793 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:28.179734    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" event={"ID":"1c84915e-e66c-406f-a253-86724e1536cf","Type":"ContainerStarted","Data":"32cdb2eefe1830224f2e160e90236d13c2538ce150e1c2368fa8379833f76d58"}
Nov 03 22:20:28.289404 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:28.289334    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" podStartSLOduration=1.289321667 podStartE2EDuration="1.289321667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:28.2884056 +0000 UTC m=+5529.919719077" watchObservedRunningTime="2026-11-03 22:20:28.289321667 +0000 UTC m=+5529.920635141"
Nov 03 22:20:39.187269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:39.187238    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:41.476959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.476926    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-679fff6fd4-krq72"]
Nov 03 22:20:41.480394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.480368    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.535233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.535201    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-679fff6fd4-krq72"]
Nov 03 22:20:41.654610 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.654571    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"]
Nov 03 22:20:41.658095 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.658078    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.659503 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.659483    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c866d43-d902-448b-b17b-89b9ef7bdb4f-plugins-config-volume\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.659598 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.659510    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfw7\" (UniqueName: \"kubernetes.io/projected/7c866d43-d902-448b-b17b-89b9ef7bdb4f-kube-api-access-gnfw7\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.714852 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.714824    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"]
Nov 03 22:20:41.760130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.760068    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2586x\" (UniqueName: \"kubernetes.io/projected/273ee43f-335f-4f07-8949-d3e33b4c55bc-kube-api-access-2586x\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.760257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.760136    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ee43f-335f-4f07-8949-d3e33b4c55bc-plugins-config-volume\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.760257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.760194    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c866d43-d902-448b-b17b-89b9ef7bdb4f-plugins-config-volume\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.760257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.760218    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfw7\" (UniqueName: \"kubernetes.io/projected/7c866d43-d902-448b-b17b-89b9ef7bdb4f-kube-api-access-gnfw7\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.760971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.760952    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c866d43-d902-448b-b17b-89b9ef7bdb4f-plugins-config-volume\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.818218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.818193    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfw7\" (UniqueName: \"kubernetes.io/projected/7c866d43-d902-448b-b17b-89b9ef7bdb4f-kube-api-access-gnfw7\") pod \"payload-processing-679fff6fd4-krq72\" (UID: \"7c866d43-d902-448b-b17b-89b9ef7bdb4f\") " pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:41.861187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.861161    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2586x\" (UniqueName: \"kubernetes.io/projected/273ee43f-335f-4f07-8949-d3e33b4c55bc-kube-api-access-2586x\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.861304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.861256    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ee43f-335f-4f07-8949-d3e33b4c55bc-plugins-config-volume\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.861928 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.861910    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/273ee43f-335f-4f07-8949-d3e33b4c55bc-plugins-config-volume\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.918618 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.918595    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2586x\" (UniqueName: \"kubernetes.io/projected/273ee43f-335f-4f07-8949-d3e33b4c55bc-kube-api-access-2586x\") pod \"payload-pre-processing-857c5f8fc8-tkszh\" (UID: \"273ee43f-335f-4f07-8949-d3e33b4c55bc\") " pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:41.970721 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:41.970696    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:42.093264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:42.093237    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:42.333871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:42.333845    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"]
Nov 03 22:20:42.335575 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:20:42.335545    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273ee43f_335f_4f07_8949_d3e33b4c55bc.slice/crio-6ad3e5245e498143e4a20b984e632c6ba6dd944f8fb934df2f636432fb9182f7 WatchSource:0}: Error finding container 6ad3e5245e498143e4a20b984e632c6ba6dd944f8fb934df2f636432fb9182f7: Status 404 returned error can't find the container with id 6ad3e5245e498143e4a20b984e632c6ba6dd944f8fb934df2f636432fb9182f7
Nov 03 22:20:42.458311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:42.458285    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-679fff6fd4-krq72"]
Nov 03 22:20:42.460141 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:20:42.460111    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c866d43_d902_448b_b17b_89b9ef7bdb4f.slice/crio-c79f2aac107c7391c46c7cd93cd1a87540dff198e940c4e9b2f672993c123985 WatchSource:0}: Error finding container c79f2aac107c7391c46c7cd93cd1a87540dff198e940c4e9b2f672993c123985: Status 404 returned error can't find the container with id c79f2aac107c7391c46c7cd93cd1a87540dff198e940c4e9b2f672993c123985
Nov 03 22:20:43.259892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.259858    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-679fff6fd4-krq72" event={"ID":"7c866d43-d902-448b-b17b-89b9ef7bdb4f","Type":"ContainerStarted","Data":"57a40fbb3ac1dc901375acc920d7ace7edc498b4cf65c355208aa7ccc7a1bd7d"}
Nov 03 22:20:43.260333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.259909    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-679fff6fd4-krq72" event={"ID":"7c866d43-d902-448b-b17b-89b9ef7bdb4f","Type":"ContainerStarted","Data":"c79f2aac107c7391c46c7cd93cd1a87540dff198e940c4e9b2f672993c123985"}
Nov 03 22:20:43.260333 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.259949    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:43.261518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.261494    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh" event={"ID":"273ee43f-335f-4f07-8949-d3e33b4c55bc","Type":"ContainerStarted","Data":"6740c149985a550171577bf2ac3da66a85f0df080cde8f61991231c571e7a453"}
Nov 03 22:20:43.261670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.261530    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh" event={"ID":"273ee43f-335f-4f07-8949-d3e33b4c55bc","Type":"ContainerStarted","Data":"6ad3e5245e498143e4a20b984e632c6ba6dd944f8fb934df2f636432fb9182f7"}
Nov 03 22:20:43.261670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.261564    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:43.371106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.371055    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-679fff6fd4-krq72" podStartSLOduration=2.371042591 podStartE2EDuration="2.371042591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:43.368985593 +0000 UTC m=+5545.000299069" watchObservedRunningTime="2026-11-03 22:20:43.371042591 +0000 UTC m=+5545.002356089"
Nov 03 22:20:43.478936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:43.478893    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh" podStartSLOduration=2.478882735 podStartE2EDuration="2.478882735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:43.477664722 +0000 UTC m=+5545.108978208" watchObservedRunningTime="2026-11-03 22:20:43.478882735 +0000 UTC m=+5545.110196250"
Nov 03 22:20:49.260403 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:20:49.260369    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:20:54.269509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:54.269480    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-679fff6fd4-krq72"
Nov 03 22:20:54.269953 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:54.269554    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-857c5f8fc8-tkszh"
Nov 03 22:20:54.428148 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:54.428112    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:20:54.515452 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:54.515420    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:20:59.428871 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.428827    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" podUID="1c84915e-e66c-406f-a253-86724e1536cf" containerName="payload-processing" containerID="cri-o://b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731" gracePeriod=25
Nov 03 22:20:59.515875 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.515833    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" podUID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" containerName="payload-pre-processing" containerID="cri-o://8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458" gracePeriod=25
Nov 03 22:20:59.759570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.759545    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:20:59.791025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.791002    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:20:59.812985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.812963    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\" (UniqueName: \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\") pod \"1c84915e-e66c-406f-a253-86724e1536cf\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") "
Nov 03 22:20:59.813103 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.813071    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\") pod \"1c84915e-e66c-406f-a253-86724e1536cf\" (UID: \"1c84915e-e66c-406f-a253-86724e1536cf\") "
Nov 03 22:20:59.813425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.813403    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume" pod "1c84915e-e66c-406f-a253-86724e1536cf" (UID: "1c84915e-e66c-406f-a253-86724e1536cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:59.814948 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.814928    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7" pod "1c84915e-e66c-406f-a253-86724e1536cf" (UID: "1c84915e-e66c-406f-a253-86724e1536cf"). InnerVolumeSpecName "kube-api-access-gx5q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:59.914425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.914394    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\") pod \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") "
Nov 03 22:20:59.914603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.914487    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\" (UniqueName: \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\") pod \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\" (UID: \"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e\") "
Nov 03 22:20:59.914750 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.914737    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c84915e-e66c-406f-a253-86724e1536cf-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.914816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.914750    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5q7\" (UniqueName: \"kubernetes.io/projected/1c84915e-e66c-406f-a253-86724e1536cf-kube-api-access-gx5q7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.914816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.914768    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume" pod "2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" (UID: "2ef7e1f1-0cb6-40ca-b3c4-382aa770823e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:59.916630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:20:59.916606    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl" pod "2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" (UID: "2ef7e1f1-0cb6-40ca-b3c4-382aa770823e"). InnerVolumeSpecName "kube-api-access-n86vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:00.016286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.016202    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86vl\" (UniqueName: \"kubernetes.io/projected/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-kube-api-access-n86vl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:21:00.016286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.016241    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:21:00.350438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.350355    5533 generic.go:356] "Generic (PLEG): container finished" podID="1c84915e-e66c-406f-a253-86724e1536cf" containerID="b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731" exitCode=0
Nov 03 22:21:00.350438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.350399    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f"
Nov 03 22:21:00.350686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.350440    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" event={"ID":"1c84915e-e66c-406f-a253-86724e1536cf","Type":"ContainerDied","Data":"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"}
Nov 03 22:21:00.350686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.350497    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-995bb6bdd-fd56f" event={"ID":"1c84915e-e66c-406f-a253-86724e1536cf","Type":"ContainerDied","Data":"32cdb2eefe1830224f2e160e90236d13c2538ce150e1c2368fa8379833f76d58"}
Nov 03 22:21:00.350686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.350530    5533 scope.go:122] "RemoveContainer" containerID="b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"
Nov 03 22:21:00.351990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.351958    5533 generic.go:356] "Generic (PLEG): container finished" podID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" containerID="8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458" exitCode=0
Nov 03 22:21:00.352117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.351999    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" event={"ID":"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e","Type":"ContainerDied","Data":"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"}
Nov 03 22:21:00.352117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.352056    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8" event={"ID":"2ef7e1f1-0cb6-40ca-b3c4-382aa770823e","Type":"ContainerDied","Data":"288bb160410efc9c0bc92be2e8055f096cc00b51acf64279d3e8be1d2c86a780"}
Nov 03 22:21:00.352117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.352005    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c65954f6c-wntm8"
Nov 03 22:21:00.359718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.359702    5533 scope.go:122] "RemoveContainer" containerID="b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"
Nov 03 22:21:00.359975 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:21:00.359952    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731\": container with ID starting with b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731 not found: ID does not exist" containerID="b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"
Nov 03 22:21:00.360088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.359984    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731"} err="failed to get container status \"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731\": rpc error: code = NotFound desc = could not find container \"b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731\": container with ID starting with b6ddeffe62db164d4236e59fb32c61f1748e6c3b9fb2d3f627074d057dc63731 not found: ID does not exist"
Nov 03 22:21:00.360088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.360001    5533 scope.go:122] "RemoveContainer" containerID="8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"
Nov 03 22:21:00.368611 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.368594    5533 scope.go:122] "RemoveContainer" containerID="8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"
Nov 03 22:21:00.368878 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:21:00.368858    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458\": container with ID starting with 8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458 not found: ID does not exist" containerID="8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"
Nov 03 22:21:00.368950 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.368884    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458"} err="failed to get container status \"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458\": rpc error: code = NotFound desc = could not find container \"8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458\": container with ID starting with 8f056d57068f24f345c879daf2fbe1e54bb58efa92d85066bf6d98fa3f8bd458 not found: ID does not exist"
Nov 03 22:21:00.507933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.507899    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:21:00.511562 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.511535    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-995bb6bdd-fd56f"]
Nov 03 22:21:00.623719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.623626    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:21:00.627689 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:00.627660    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-c65954f6c-wntm8"]
Nov 03 22:21:01.388565 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.388525    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:21:01.388962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.388946    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1c84915e-e66c-406f-a253-86724e1536cf" containerName="payload-processing"
Nov 03 22:21:01.388962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.388962    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c84915e-e66c-406f-a253-86724e1536cf" containerName="payload-processing"
Nov 03 22:21:01.389097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.388983    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" containerName="payload-pre-processing"
Nov 03 22:21:01.389097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.388988    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" containerName="payload-pre-processing"
Nov 03 22:21:01.389097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.389048    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c84915e-e66c-406f-a253-86724e1536cf" containerName="payload-processing"
Nov 03 22:21:01.389097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.389057    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" containerName="payload-pre-processing"
Nov 03 22:21:01.393517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.393494    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.430389 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.430355    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.430519 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.430400    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzhz\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.430593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.430516    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.430593 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.430559    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.437062 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.437028    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:21:01.482124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.481425    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 22:21:01.482124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.481434    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 22:21:01.482124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.481477    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 22:21:01.482124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.481565    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 22:21:01.489592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.489566    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 22:21:01.489592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.489586    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pp8qc\""
Nov 03 22:21:01.531323 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.531300    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.531678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.531339    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.531678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.531431    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.531678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.531466    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzhz\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.532114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.532086    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.532218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.532178    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.533646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.533622    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.589668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.589626    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzhz\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\") pod \"payload-processing-external-model-7fd6967f4c-lp5b8\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.708190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.708161    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:01.964051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:01.963974    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:21:02.076202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.076075    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:21:02.078528 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:21:02.078499    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca867c14_e082_400e_9c40_dbc0381c5e41.slice/crio-69f5db16edf9672204db89088a1b431f77d146327933b199ec59b313d7f8c773 WatchSource:0}: Error finding container 69f5db16edf9672204db89088a1b431f77d146327933b199ec59b313d7f8c773: Status 404 returned error can't find the container with id 69f5db16edf9672204db89088a1b431f77d146327933b199ec59b313d7f8c773
Nov 03 22:21:02.368510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.368413    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" event={"ID":"ca867c14-e082-400e-9c40-dbc0381c5e41","Type":"ContainerStarted","Data":"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"}
Nov 03 22:21:02.368510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.368480    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:02.368510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.368501    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" event={"ID":"ca867c14-e082-400e-9c40-dbc0381c5e41","Type":"ContainerStarted","Data":"69f5db16edf9672204db89088a1b431f77d146327933b199ec59b313d7f8c773"}
Nov 03 22:21:02.480878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.480798    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" podStartSLOduration=1.480780441 podStartE2EDuration="1.480780441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:02.477870367 +0000 UTC m=+5564.109183843" watchObservedRunningTime="2026-11-03 22:21:02.480780441 +0000 UTC m=+5564.112093919"
Nov 03 22:21:02.495023 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.494977    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c84915e-e66c-406f-a253-86724e1536cf" path="/var/lib/kubelet/pods/1c84915e-e66c-406f-a253-86724e1536cf/volumes"
Nov 03 22:21:02.495583 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:02.495553    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef7e1f1-0cb6-40ca-b3c4-382aa770823e" path="/var/lib/kubelet/pods/2ef7e1f1-0cb6-40ca-b3c4-382aa770823e/volumes"
Nov 03 22:21:13.388241 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:13.388211    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:21:19.260096 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:21:19.260062    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:21:22.645387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:22.645354    5533 scope.go:122] "RemoveContainer" containerID="2c388c9725d2b3367c24529d9dabba8bfc4b9cdcbe967f1bfa4a5cbca0f44368"
Nov 03 22:21:22.654267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:22.654247    5533 scope.go:122] "RemoveContainer" containerID="e5f686b4b65debdbef92b45def41cdb388945e2a57e0e9c630179d056913dda7"
Nov 03 22:21:26.045158 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:26.045120    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"]
Nov 03 22:21:26.091589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:26.091558    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9164cdf1-rkrff"]
Nov 03 22:21:26.498506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:21:26.498476    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1decd0f1-6b66-4777-984a-72662e3f2d99" path="/var/lib/kubelet/pods/1decd0f1-6b66-4777-984a-72662e3f2d99/volumes"
Nov 03 22:21:49.260608 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:21:49.260569    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:22:19.260575 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:22:19.260540    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:22:45.196337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.196303    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"]
Nov 03 22:22:45.199745 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.199724    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.205862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.205836    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.205997 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.205935    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.206076 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.206044    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbrp\" (UniqueName: \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.249019 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.248997    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:45.295896 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.295875    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:22:45.297886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.297868    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"]
Nov 03 22:22:45.306989 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.306960    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbrp\" (UniqueName: \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.307121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.307003    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.307121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.307056    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.307827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.307804    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.309601 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.309576    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.365003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.364983    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbrp\" (UniqueName: \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\") pod \"maas-api-revoke-sub-e2547622-krktf\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.512013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.511964    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:45.875932 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.875902    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"]
Nov 03 22:22:45.877910 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:22:45.877882    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01a5fb1_9185_4568_9a6f_5a3f49341a63.slice/crio-38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3 WatchSource:0}: Error finding container 38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3: Status 404 returned error can't find the container with id 38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3
Nov 03 22:22:45.934284 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:45.934255    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf" event={"ID":"b01a5fb1-9185-4568-9a6f-5a3f49341a63","Type":"ContainerStarted","Data":"38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3"}
Nov 03 22:22:46.940534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:46.940502    5533 generic.go:356] "Generic (PLEG): container finished" podID="b01a5fb1-9185-4568-9a6f-5a3f49341a63" containerID="1963a2d2897bacff3bf5ea5494e57f4073b7f621dbbe41cf9d0214ac3775c83e" exitCode=0
Nov 03 22:22:46.940904 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:46.940536    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf" event={"ID":"b01a5fb1-9185-4568-9a6f-5a3f49341a63","Type":"ContainerDied","Data":"1963a2d2897bacff3bf5ea5494e57f4073b7f621dbbe41cf9d0214ac3775c83e"}
Nov 03 22:22:48.074814 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.074788    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:48.132227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.132196    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\") pod \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") "
Nov 03 22:22:48.132375 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.132296    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\" (UniqueName: \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\") pod \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") "
Nov 03 22:22:48.132375 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.132348    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\") pod \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\" (UID: \"b01a5fb1-9185-4568-9a6f-5a3f49341a63\") "
Nov 03 22:22:48.132541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.132517    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca" pod "b01a5fb1-9185-4568-9a6f-5a3f49341a63" (UID: "b01a5fb1-9185-4568-9a6f-5a3f49341a63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:48.132629 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.132607    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:22:48.134254 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.134232    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp" pod "b01a5fb1-9185-4568-9a6f-5a3f49341a63" (UID: "b01a5fb1-9185-4568-9a6f-5a3f49341a63"). InnerVolumeSpecName "kube-api-access-mnbrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:48.134330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.134310    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca" pod "b01a5fb1-9185-4568-9a6f-5a3f49341a63" (UID: "b01a5fb1-9185-4568-9a6f-5a3f49341a63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:48.233774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.233718    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbrp\" (UniqueName: \"kubernetes.io/projected/b01a5fb1-9185-4568-9a6f-5a3f49341a63-kube-api-access-mnbrp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:22:48.233774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.233740    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01a5fb1-9185-4568-9a6f-5a3f49341a63-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:22:48.952097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.952065    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf" event={"ID":"b01a5fb1-9185-4568-9a6f-5a3f49341a63","Type":"ContainerDied","Data":"38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3"}
Nov 03 22:22:48.952231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.952116    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38891afe7909318797951085c0bac4a5816da948ca7f87afc2cdd2872f07a1a3"
Nov 03 22:22:48.952231 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:48.952073    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"
Nov 03 22:22:49.260331 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:22:49.260263    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:22:58.642025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:58.641992    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:22:59.904834 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:22:59.904748    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:23:03.643257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.643212    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" podUID="ca867c14-e082-400e-9c40-dbc0381c5e41" containerName="payload-processing" containerID="cri-o://63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a" gracePeriod=25
Nov 03 22:23:03.890290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.890264    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:23:03.966620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.966595    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\") pod \"ca867c14-e082-400e-9c40-dbc0381c5e41\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") "
Nov 03 22:23:03.966777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.966678    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\") pod \"ca867c14-e082-400e-9c40-dbc0381c5e41\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") "
Nov 03 22:23:03.966777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.966729    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\") pod \"ca867c14-e082-400e-9c40-dbc0381c5e41\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") "
Nov 03 22:23:03.966777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.966766    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\") pod \"ca867c14-e082-400e-9c40-dbc0381c5e41\" (UID: \"ca867c14-e082-400e-9c40-dbc0381c5e41\") "
Nov 03 22:23:03.967172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.967148    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume" pod "ca867c14-e082-400e-9c40-dbc0381c5e41" (UID: "ca867c14-e082-400e-9c40-dbc0381c5e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:03.967172 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.967158    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume" pod "ca867c14-e082-400e-9c40-dbc0381c5e41" (UID: "ca867c14-e082-400e-9c40-dbc0381c5e41"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:03.968717 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.968692    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials" pod "ca867c14-e082-400e-9c40-dbc0381c5e41" (UID: "ca867c14-e082-400e-9c40-dbc0381c5e41"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:03.968813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:03.968698    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz" pod "ca867c14-e082-400e-9c40-dbc0381c5e41" (UID: "ca867c14-e082-400e-9c40-dbc0381c5e41"). InnerVolumeSpecName "kube-api-access-6hzhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:04.043803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.043776    5533 generic.go:356] "Generic (PLEG): container finished" podID="ca867c14-e082-400e-9c40-dbc0381c5e41" containerID="63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a" exitCode=0
Nov 03 22:23:04.043935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.043824    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"
Nov 03 22:23:04.043935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.043865    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" event={"ID":"ca867c14-e082-400e-9c40-dbc0381c5e41","Type":"ContainerDied","Data":"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"}
Nov 03 22:23:04.043935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.043918    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8" event={"ID":"ca867c14-e082-400e-9c40-dbc0381c5e41","Type":"ContainerDied","Data":"69f5db16edf9672204db89088a1b431f77d146327933b199ec59b313d7f8c773"}
Nov 03 22:23:04.044059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.043940    5533 scope.go:122] "RemoveContainer" containerID="63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"
Nov 03 22:23:04.053030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.053012    5533 scope.go:122] "RemoveContainer" containerID="63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"
Nov 03 22:23:04.053307 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:23:04.053288    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a\": container with ID starting with 63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a not found: ID does not exist" containerID="63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"
Nov 03 22:23:04.053353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.053317    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a"} err="failed to get container status \"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a\": rpc error: code = NotFound desc = could not find container \"63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a\": container with ID starting with 63ca003512224c8aa5f9222581863179c8ce76f276c3d8d0b0cd68528de9b95a not found: ID does not exist"
Nov 03 22:23:04.068191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.068163    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:04.068191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.068184    5533 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca867c14-e082-400e-9c40-dbc0381c5e41-routing-overlay-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:04.068191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.068193    5533 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-provider-credentials\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:04.068428 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.068203    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzhz\" (UniqueName: \"kubernetes.io/projected/ca867c14-e082-400e-9c40-dbc0381c5e41-kube-api-access-6hzhz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:04.198194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.198172    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:23:04.204849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.204819    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-lp5b8"]
Nov 03 22:23:04.491647 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:04.491604    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca867c14-e082-400e-9c40-dbc0381c5e41" path="/var/lib/kubelet/pods/ca867c14-e082-400e-9c40-dbc0381c5e41/volumes"
Nov 03 22:23:10.723381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:10.723342    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:23:16.377712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.377672    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"]
Nov 03 22:23:16.378108 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378068    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b01a5fb1-9185-4568-9a6f-5a3f49341a63" containerName="revoke-keys"
Nov 03 22:23:16.378108 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378081    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01a5fb1-9185-4568-9a6f-5a3f49341a63" containerName="revoke-keys"
Nov 03 22:23:16.378187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378112    5533 cpu_manager.go:414] "containerMap: removing container" podUID="ca867c14-e082-400e-9c40-dbc0381c5e41" containerName="payload-processing"
Nov 03 22:23:16.378187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378118    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca867c14-e082-400e-9c40-dbc0381c5e41" containerName="payload-processing"
Nov 03 22:23:16.378187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378181    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01a5fb1-9185-4568-9a6f-5a3f49341a63" containerName="revoke-keys"
Nov 03 22:23:16.378294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.378194    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca867c14-e082-400e-9c40-dbc0381c5e41" containerName="payload-processing"
Nov 03 22:23:16.381336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.381311    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.479480 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.479450    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:16.479687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.479530    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:23:16.481669 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.481627    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"]
Nov 03 22:23:16.494794 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.494768    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.494922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.494800    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxng\" (UniqueName: \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.494922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.494818    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.595779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.595749    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.595779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.595783    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxng\" (UniqueName: \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.596006 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.595803    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.596548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.596524    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.598116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.598095    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.653803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.653727    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxng\" (UniqueName: \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\") pod \"maas-api-revoke-sub-c1e271a9-dx897\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:16.695228 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:16.695204    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:17.059668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:17.059614    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"]
Nov 03 22:23:17.061964 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:23:17.061938    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded5521_27a6_4410_a172_61baf7b40e5e.slice/crio-cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4 WatchSource:0}: Error finding container cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4: Status 404 returned error can't find the container with id cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4
Nov 03 22:23:17.145035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:17.145007    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897" event={"ID":"2ded5521-27a6-4410-a172-61baf7b40e5e","Type":"ContainerStarted","Data":"cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4"}
Nov 03 22:23:18.152137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:18.152104    5533 generic.go:356] "Generic (PLEG): container finished" podID="2ded5521-27a6-4410-a172-61baf7b40e5e" containerID="d836698ddd27a7490f3800a286b403f9a8648861ff4008ae7402a20ee1c306ed" exitCode=0
Nov 03 22:23:18.152586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:18.152157    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897" event={"ID":"2ded5521-27a6-4410-a172-61baf7b40e5e","Type":"ContainerDied","Data":"d836698ddd27a7490f3800a286b403f9a8648861ff4008ae7402a20ee1c306ed"}
Nov 03 22:23:19.264506 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:23:19.264470    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:23:19.296521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.296498    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:19.420403 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.420381    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\" (UniqueName: \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\") pod \"2ded5521-27a6-4410-a172-61baf7b40e5e\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") "
Nov 03 22:23:19.420546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.420435    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\") pod \"2ded5521-27a6-4410-a172-61baf7b40e5e\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") "
Nov 03 22:23:19.420546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.420469    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\") pod \"2ded5521-27a6-4410-a172-61baf7b40e5e\" (UID: \"2ded5521-27a6-4410-a172-61baf7b40e5e\") "
Nov 03 22:23:19.420753 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.420730    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca" pod "2ded5521-27a6-4410-a172-61baf7b40e5e" (UID: "2ded5521-27a6-4410-a172-61baf7b40e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:19.420843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.420824    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:19.422410 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.422383    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca" pod "2ded5521-27a6-4410-a172-61baf7b40e5e" (UID: "2ded5521-27a6-4410-a172-61baf7b40e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:19.422514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.422490    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng" pod "2ded5521-27a6-4410-a172-61baf7b40e5e" (UID: "2ded5521-27a6-4410-a172-61baf7b40e5e"). InnerVolumeSpecName "kube-api-access-zlxng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:19.517922 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.517902    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:23:19.521230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.521211    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxng\" (UniqueName: \"kubernetes.io/projected/2ded5521-27a6-4410-a172-61baf7b40e5e-kube-api-access-zlxng\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:19.521230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.521231    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ded5521-27a6-4410-a172-61baf7b40e5e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:19.535226 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.535204    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:23:19.572267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.572248    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:23:19.582851 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:19.582832    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:23:20.165065 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:20.165025    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897" event={"ID":"2ded5521-27a6-4410-a172-61baf7b40e5e","Type":"ContainerDied","Data":"cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4"}
Nov 03 22:23:20.165235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:20.165073    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"
Nov 03 22:23:20.165235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:20.165117    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbb91455ee7831fc8f353a826ba297d3c8ae791bf7edccc4fef2dbea03345ed4"
Nov 03 22:23:22.693850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:22.693817    5533 scope.go:122] "RemoveContainer" containerID="7398cffbc1f9e404524e0c7a7a09390365edc8178654fbbab4cef8703d81bb86"
Nov 03 22:23:29.262591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:29.262555    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:23:40.205474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:40.205441    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:23:41.816215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.816184    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"]
Nov 03 22:23:41.816605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.816587    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2ded5521-27a6-4410-a172-61baf7b40e5e" containerName="revoke-keys"
Nov 03 22:23:41.816605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.816601    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ded5521-27a6-4410-a172-61baf7b40e5e" containerName="revoke-keys"
Nov 03 22:23:41.816716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.816708    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ded5521-27a6-4410-a172-61baf7b40e5e" containerName="revoke-keys"
Nov 03 22:23:41.819838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.819820    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.857064 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.857039    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmj8\" (UniqueName: \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.857187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.857095    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.857234 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.857186    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.869214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.869195    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:41.915846 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.915818    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:23:41.917898 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.917875    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"]
Nov 03 22:23:41.957752 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.957721    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.957899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.957794    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.957970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.957902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmj8\" (UniqueName: \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.958568 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.958544    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:41.960290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:41.960264    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:42.016085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:42.016057    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmj8\" (UniqueName: \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\") pod \"maas-api-revoke-sub-2e1f5b5f-fx2pz\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:42.132032 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:42.131958    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:42.497229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:42.497194    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"]
Nov 03 22:23:42.501872 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:23:42.501842    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b4556b_585f_4085_bcd1_8787056f1f04.slice/crio-297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41 WatchSource:0}: Error finding container 297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41: Status 404 returned error can't find the container with id 297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41
Nov 03 22:23:43.295928 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:43.295897    5533 generic.go:356] "Generic (PLEG): container finished" podID="55b4556b-585f-4085-bcd1-8787056f1f04" containerID="fa2b4b1e737dda8998fe5ce6db10c4abced9784cce2db3ca0c312ec764352a7f" exitCode=0
Nov 03 22:23:43.296286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:43.295952    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz" event={"ID":"55b4556b-585f-4085-bcd1-8787056f1f04","Type":"ContainerDied","Data":"fa2b4b1e737dda8998fe5ce6db10c4abced9784cce2db3ca0c312ec764352a7f"}
Nov 03 22:23:43.296286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:43.295989    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz" event={"ID":"55b4556b-585f-4085-bcd1-8787056f1f04","Type":"ContainerStarted","Data":"297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41"}
Nov 03 22:23:44.430711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.430680    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:44.478729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.478705    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\") pod \"55b4556b-585f-4085-bcd1-8787056f1f04\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") "
Nov 03 22:23:44.478862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.478815    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\") pod \"55b4556b-585f-4085-bcd1-8787056f1f04\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") "
Nov 03 22:23:44.478862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.478846    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\" (UniqueName: \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\") pod \"55b4556b-585f-4085-bcd1-8787056f1f04\" (UID: \"55b4556b-585f-4085-bcd1-8787056f1f04\") "
Nov 03 22:23:44.479056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.479033    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca" pod "55b4556b-585f-4085-bcd1-8787056f1f04" (UID: "55b4556b-585f-4085-bcd1-8787056f1f04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:44.480977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.480949    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca" pod "55b4556b-585f-4085-bcd1-8787056f1f04" (UID: "55b4556b-585f-4085-bcd1-8787056f1f04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:44.481069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.480983    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8" pod "55b4556b-585f-4085-bcd1-8787056f1f04" (UID: "55b4556b-585f-4085-bcd1-8787056f1f04"). InnerVolumeSpecName "kube-api-access-frmj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:44.580350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.580298    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:44.580350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.580321    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frmj8\" (UniqueName: \"kubernetes.io/projected/55b4556b-585f-4085-bcd1-8787056f1f04-kube-api-access-frmj8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:44.580350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:44.580332    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b4556b-585f-4085-bcd1-8787056f1f04-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:45.308054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:45.308022    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"
Nov 03 22:23:45.308054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:45.308031    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz" event={"ID":"55b4556b-585f-4085-bcd1-8787056f1f04","Type":"ContainerDied","Data":"297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41"}
Nov 03 22:23:45.308272 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:45.308077    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="297a9050581df502332919bf76a6ffa5485b68bc0c815352f134f15bc3cd2b41"
Nov 03 22:23:49.261698 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:23:49.261658    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:23:54.291819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.291779    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"]
Nov 03 22:23:54.292279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.292174    5533 cpu_manager.go:414] "containerMap: removing container" podUID="55b4556b-585f-4085-bcd1-8787056f1f04" containerName="revoke-keys"
Nov 03 22:23:54.292279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.292185    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b4556b-585f-4085-bcd1-8787056f1f04" containerName="revoke-keys"
Nov 03 22:23:54.292279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.292243    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b4556b-585f-4085-bcd1-8787056f1f04" containerName="revoke-keys"
Nov 03 22:23:54.295717 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.295698    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.361936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.361900    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.362103 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.361951    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6vn\" (UniqueName: \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.362103 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.362071    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.393782 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.393753    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:23:54.393910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.393795    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:23:54.395958 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.395938    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"]
Nov 03 22:23:54.463264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.463239    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.463385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.463267    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6vn\" (UniqueName: \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.463385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.463294    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.463906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.463889    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.465613 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.465584    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.520919 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.520897    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6vn\" (UniqueName: \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\") pod \"maas-api-revoke-sub-ec843018-2bnh4\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.611038 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.610977    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:54.983058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:54.983031    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"]
Nov 03 22:23:54.985207 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:23:54.985176    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363093ed_6a0e_43ee_b7ec_fd162bc3abba.slice/crio-4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116 WatchSource:0}: Error finding container 4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116: Status 404 returned error can't find the container with id 4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116
Nov 03 22:23:55.359299 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:55.359265    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4" event={"ID":"363093ed-6a0e-43ee-b7ec-fd162bc3abba","Type":"ContainerStarted","Data":"830d26deca06e4c127a44db792778d7f039cad99b003c9ae93df35ca722f8352"}
Nov 03 22:23:55.359683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:55.359317    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4" event={"ID":"363093ed-6a0e-43ee-b7ec-fd162bc3abba","Type":"ContainerStarted","Data":"4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116"}
Nov 03 22:23:55.468734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:55.468678    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4" podStartSLOduration=1.468661916 podStartE2EDuration="1.468661916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:55.467778322 +0000 UTC m=+5737.099091824" watchObservedRunningTime="2026-11-03 22:23:55.468661916 +0000 UTC m=+5737.099975394"
Nov 03 22:23:56.365858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:56.365821    5533 generic.go:356] "Generic (PLEG): container finished" podID="363093ed-6a0e-43ee-b7ec-fd162bc3abba" containerID="830d26deca06e4c127a44db792778d7f039cad99b003c9ae93df35ca722f8352" exitCode=0
Nov 03 22:23:56.366258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:56.365922    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4" event={"ID":"363093ed-6a0e-43ee-b7ec-fd162bc3abba","Type":"ContainerDied","Data":"830d26deca06e4c127a44db792778d7f039cad99b003c9ae93df35ca722f8352"}
Nov 03 22:23:57.501312 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.501288    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:57.591304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.591278    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\") pod \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") "
Nov 03 22:23:57.591453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.591329    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\") pod \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") "
Nov 03 22:23:57.591453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.591382    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\" (UniqueName: \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\") pod \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\" (UID: \"363093ed-6a0e-43ee-b7ec-fd162bc3abba\") "
Nov 03 22:23:57.591725 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.591693    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca" pod "363093ed-6a0e-43ee-b7ec-fd162bc3abba" (UID: "363093ed-6a0e-43ee-b7ec-fd162bc3abba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:57.593476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.593455    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn" pod "363093ed-6a0e-43ee-b7ec-fd162bc3abba" (UID: "363093ed-6a0e-43ee-b7ec-fd162bc3abba"). InnerVolumeSpecName "kube-api-access-8h6vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:57.593569 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.593476    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca" pod "363093ed-6a0e-43ee-b7ec-fd162bc3abba" (UID: "363093ed-6a0e-43ee-b7ec-fd162bc3abba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:57.692264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.692236    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:57.692264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.692261    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6vn\" (UniqueName: \"kubernetes.io/projected/363093ed-6a0e-43ee-b7ec-fd162bc3abba-kube-api-access-8h6vn\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:57.692440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:57.692273    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363093ed-6a0e-43ee-b7ec-fd162bc3abba-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:23:58.378425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:58.378396    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"
Nov 03 22:23:58.378425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:58.378411    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4" event={"ID":"363093ed-6a0e-43ee-b7ec-fd162bc3abba","Type":"ContainerDied","Data":"4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116"}
Nov 03 22:23:58.378668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:58.378460    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4806b929782085500c2a05e15f755643f65e2e105147b738f84a8a421b1c3116"
Nov 03 22:23:58.395660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:23:58.395571    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:24:02.045725 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:02.045688    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"]
Nov 03 22:24:02.092467 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:02.092439    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5x9p2"]
Nov 03 22:24:02.491532 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:02.491500    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523a5f01-e51d-470d-99cf-6fa730d61d30" path="/var/lib/kubelet/pods/523a5f01-e51d-470d-99cf-6fa730d61d30/volumes"
Nov 03 22:24:11.509160 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:11.509127    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:24:19.260356 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:24:19.260317    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:24:22.719795 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:22.719762    5533 scope.go:122] "RemoveContainer" containerID="beff191f2afd23a0b45f934a04e8cf9b7344e8e386a5a4919f68cbc19ce4d090"
Nov 03 22:24:29.224075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.223997    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"]
Nov 03 22:24:29.224542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.224389    5533 cpu_manager.go:414] "containerMap: removing container" podUID="363093ed-6a0e-43ee-b7ec-fd162bc3abba" containerName="revoke-keys"
Nov 03 22:24:29.224542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.224401    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363093ed-6a0e-43ee-b7ec-fd162bc3abba" containerName="revoke-keys"
Nov 03 22:24:29.224542 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.224460    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="363093ed-6a0e-43ee-b7ec-fd162bc3abba" containerName="revoke-keys"
Nov 03 22:24:29.227686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.227670    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.270085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.270059    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.270206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.270111    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.270254 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.270232    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh44\" (UniqueName: \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.279072 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.279039    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:24:29.325827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.325795    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:24:29.328040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.328011    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"]
Nov 03 22:24:29.371115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.371088    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh44\" (UniqueName: \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.371232 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.371130    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.371232 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.371183    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.371906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.371884    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.373656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.373606    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.428799 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.428772    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh44\" (UniqueName: \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\") pod \"maas-api-revoke-sub-046e1343-zqkfn\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.539819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.539771    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:29.904645 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:29.904605    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"]
Nov 03 22:24:29.906285 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:24:29.906209    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bbe15e_9623_408a_a877_20d54c91c8f3.slice/crio-0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8 WatchSource:0}: Error finding container 0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8: Status 404 returned error can't find the container with id 0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8
Nov 03 22:24:30.564296 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:30.564261    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn" event={"ID":"83bbe15e-9623-408a-a877-20d54c91c8f3","Type":"ContainerStarted","Data":"615ae8e10088a3fb7f45b44e9a42afddd99413951ac6958bdcc74f10caae74bc"}
Nov 03 22:24:30.564702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:30.564315    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn" event={"ID":"83bbe15e-9623-408a-a877-20d54c91c8f3","Type":"ContainerStarted","Data":"0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8"}
Nov 03 22:24:30.673554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:30.673504    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn" podStartSLOduration=1.673489806 podStartE2EDuration="1.673489806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:30.672533869 +0000 UTC m=+5772.303847346" watchObservedRunningTime="2026-11-03 22:24:30.673489806 +0000 UTC m=+5772.304803283"
Nov 03 22:24:31.570767 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:31.570736    5533 generic.go:356] "Generic (PLEG): container finished" podID="83bbe15e-9623-408a-a877-20d54c91c8f3" containerID="615ae8e10088a3fb7f45b44e9a42afddd99413951ac6958bdcc74f10caae74bc" exitCode=0
Nov 03 22:24:31.571186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:31.570821    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn" event={"ID":"83bbe15e-9623-408a-a877-20d54c91c8f3","Type":"ContainerDied","Data":"615ae8e10088a3fb7f45b44e9a42afddd99413951ac6958bdcc74f10caae74bc"}
Nov 03 22:24:32.709109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.709085    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:32.799804 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.799774    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\" (UniqueName: \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\") pod \"83bbe15e-9623-408a-a877-20d54c91c8f3\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") "
Nov 03 22:24:32.799929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.799831    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\") pod \"83bbe15e-9623-408a-a877-20d54c91c8f3\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") "
Nov 03 22:24:32.799929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.799892    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\") pod \"83bbe15e-9623-408a-a877-20d54c91c8f3\" (UID: \"83bbe15e-9623-408a-a877-20d54c91c8f3\") "
Nov 03 22:24:32.800278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.800248    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca" pod "83bbe15e-9623-408a-a877-20d54c91c8f3" (UID: "83bbe15e-9623-408a-a877-20d54c91c8f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:24:32.801708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.801684    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44" pod "83bbe15e-9623-408a-a877-20d54c91c8f3" (UID: "83bbe15e-9623-408a-a877-20d54c91c8f3"). InnerVolumeSpecName "kube-api-access-9hh44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:24:32.801904 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.801879    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca" pod "83bbe15e-9623-408a-a877-20d54c91c8f3" (UID: "83bbe15e-9623-408a-a877-20d54c91c8f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:24:32.901143 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.901079    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:32.901143 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.901104    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh44\" (UniqueName: \"kubernetes.io/projected/83bbe15e-9623-408a-a877-20d54c91c8f3-kube-api-access-9hh44\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:32.901143 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:32.901114    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83bbe15e-9623-408a-a877-20d54c91c8f3-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:33.583475 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:33.583439    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn" event={"ID":"83bbe15e-9623-408a-a877-20d54c91c8f3","Type":"ContainerDied","Data":"0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8"}
Nov 03 22:24:33.583663 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:33.583481    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"
Nov 03 22:24:33.583663 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:33.583488    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9ebd6681df610f6977ee24843b9a5347a1457f39ec8f4621d9008f9ce7d3f8"
Nov 03 22:24:34.049019 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:34.048984    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"]
Nov 03 22:24:34.095596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:34.095569    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2gxw9"]
Nov 03 22:24:34.491786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:34.491758    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc556fc-7177-4d40-9ae7-557c84257a3a" path="/var/lib/kubelet/pods/6cc556fc-7177-4d40-9ae7-557c84257a3a/volumes"
Nov 03 22:24:40.039931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.039899    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"]
Nov 03 22:24:40.040394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.040256    5533 cpu_manager.go:414] "containerMap: removing container" podUID="83bbe15e-9623-408a-a877-20d54c91c8f3" containerName="revoke-keys"
Nov 03 22:24:40.040394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.040267    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bbe15e-9623-408a-a877-20d54c91c8f3" containerName="revoke-keys"
Nov 03 22:24:40.040394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.040352    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bbe15e-9623-408a-a877-20d54c91c8f3" containerName="revoke-keys"
Nov 03 22:24:40.043314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.043295    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.139030 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.139002    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:24:40.139194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.139027    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:24:40.141155 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.141137    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"]
Nov 03 22:24:40.160380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.160354    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fcl\" (UniqueName: \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.160539 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.160394    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.160625 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.160602    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.261872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.261840    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.262052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.261918    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.262052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.261980    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fcl\" (UniqueName: \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.262582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.262559    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.264136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.264114    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.320774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.320723    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fcl\" (UniqueName: \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\") pod \"maas-api-revoke-sub-d3fba056-ccmnz\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.356356 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.356335    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:40.722549 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:40.722525    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"]
Nov 03 22:24:40.725419 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:24:40.725390    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe29570_6b24_4710_9581_792ec762431d.slice/crio-80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98 WatchSource:0}: Error finding container 80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98: Status 404 returned error can't find the container with id 80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98
Nov 03 22:24:41.625874 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:41.625842    5533 generic.go:356] "Generic (PLEG): container finished" podID="2fe29570-6b24-4710-9581-792ec762431d" containerID="926d6a40f451717575a0aa0c3b567d94c82d388426463a712ad9d30e8bd70ecf" exitCode=0
Nov 03 22:24:41.626193 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:41.625896    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz" event={"ID":"2fe29570-6b24-4710-9581-792ec762431d","Type":"ContainerDied","Data":"926d6a40f451717575a0aa0c3b567d94c82d388426463a712ad9d30e8bd70ecf"}
Nov 03 22:24:41.626193 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:41.625935    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz" event={"ID":"2fe29570-6b24-4710-9581-792ec762431d","Type":"ContainerStarted","Data":"80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98"}
Nov 03 22:24:42.200301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.200268    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:24:42.770535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.770510    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:42.886107 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.886007    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\") pod \"2fe29570-6b24-4710-9581-792ec762431d\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") "
Nov 03 22:24:42.886320 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.886104    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\" (UniqueName: \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\") pod \"2fe29570-6b24-4710-9581-792ec762431d\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") "
Nov 03 22:24:42.886320 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.886210    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\") pod \"2fe29570-6b24-4710-9581-792ec762431d\" (UID: \"2fe29570-6b24-4710-9581-792ec762431d\") "
Nov 03 22:24:42.886892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.886850    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca" pod "2fe29570-6b24-4710-9581-792ec762431d" (UID: "2fe29570-6b24-4710-9581-792ec762431d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:24:42.888914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.888872    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca" pod "2fe29570-6b24-4710-9581-792ec762431d" (UID: "2fe29570-6b24-4710-9581-792ec762431d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:24:42.897137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.897108    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl" pod "2fe29570-6b24-4710-9581-792ec762431d" (UID: "2fe29570-6b24-4710-9581-792ec762431d"). InnerVolumeSpecName "kube-api-access-86fcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:24:42.987541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.987493    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe29570-6b24-4710-9581-792ec762431d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:42.987541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.987531    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fcl\" (UniqueName: \"kubernetes.io/projected/2fe29570-6b24-4710-9581-792ec762431d-kube-api-access-86fcl\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:42.987541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:42.987541    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe29570-6b24-4710-9581-792ec762431d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:24:43.638861 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:43.638832    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"
Nov 03 22:24:43.639040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:43.638821    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz" event={"ID":"2fe29570-6b24-4710-9581-792ec762431d","Type":"ContainerDied","Data":"80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98"}
Nov 03 22:24:43.639040 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:43.638965    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e4266d9e08c0a52ce5c350f8331b8d304942893644ac6ff8632efe4c7a4d98"
Nov 03 22:24:49.260057 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:24:49.260016    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:24:52.795685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:24:52.795646    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:00.047415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:00.047367    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"]
Nov 03 22:25:00.093993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:00.093961    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8feec754-wxstt"]
Nov 03 22:25:00.495031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:00.494983    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7122e4b5-267b-49a7-aeb7-5302dc1884e9" path="/var/lib/kubelet/pods/7122e4b5-267b-49a7-aeb7-5302dc1884e9/volumes"
Nov 03 22:25:05.482938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:05.482907    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:14.686054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.686013    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"]
Nov 03 22:25:14.686586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.686565    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2fe29570-6b24-4710-9581-792ec762431d" containerName="revoke-keys"
Nov 03 22:25:14.686713 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.686589    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe29570-6b24-4710-9581-792ec762431d" containerName="revoke-keys"
Nov 03 22:25:14.686787 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.686756    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe29570-6b24-4710-9581-792ec762431d" containerName="revoke-keys"
Nov 03 22:25:14.690062 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.690038    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.742166 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.742135    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:25:14.744822 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.744790    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"]
Nov 03 22:25:14.809469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.809447    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.809617 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.809487    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnzz\" (UniqueName: \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.809617 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.809574    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.842078 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.842053    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:25:14.910454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.910431    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.910556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.910475    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnzz\" (UniqueName: \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.910556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.910522    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.911244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.911225    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.912938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.912918    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:14.968298 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:14.968240    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnzz\" (UniqueName: \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\") pod \"maas-api-revoke-sub-ae594979-d6rsl\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:15.003047 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:15.003027    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:15.366146 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:15.366123    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"]
Nov 03 22:25:15.368243 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:25:15.368213    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d21e97_52b7_48f2_b3f9_60d476aa5680.slice/crio-0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855 WatchSource:0}: Error finding container 0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855: Status 404 returned error can't find the container with id 0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855
Nov 03 22:25:15.817406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:15.817373    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl" event={"ID":"87d21e97-52b7-48f2-b3f9-60d476aa5680","Type":"ContainerStarted","Data":"dc691aaaa360e7f0fbc819b56528f03fe111bb3a165a403029edcfea35266aa6"}
Nov 03 22:25:15.817780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:15.817428    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl" event={"ID":"87d21e97-52b7-48f2-b3f9-60d476aa5680","Type":"ContainerStarted","Data":"0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855"}
Nov 03 22:25:15.928390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:15.928347    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl" podStartSLOduration=1.928333702 podStartE2EDuration="1.928333702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:15.92589242 +0000 UTC m=+5817.557205898" watchObservedRunningTime="2026-11-03 22:25:15.928333702 +0000 UTC m=+5817.559647224"
Nov 03 22:25:16.824267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:16.824227    5533 generic.go:356] "Generic (PLEG): container finished" podID="87d21e97-52b7-48f2-b3f9-60d476aa5680" containerID="dc691aaaa360e7f0fbc819b56528f03fe111bb3a165a403029edcfea35266aa6" exitCode=0
Nov 03 22:25:16.824707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:16.824309    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl" event={"ID":"87d21e97-52b7-48f2-b3f9-60d476aa5680","Type":"ContainerDied","Data":"dc691aaaa360e7f0fbc819b56528f03fe111bb3a165a403029edcfea35266aa6"}
Nov 03 22:25:17.963811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:17.963785    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:18.037440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.037399    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\") pod \"87d21e97-52b7-48f2-b3f9-60d476aa5680\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") "
Nov 03 22:25:18.037583 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.037464    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\" (UniqueName: \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\") pod \"87d21e97-52b7-48f2-b3f9-60d476aa5680\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") "
Nov 03 22:25:18.037583 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.037522    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\") pod \"87d21e97-52b7-48f2-b3f9-60d476aa5680\" (UID: \"87d21e97-52b7-48f2-b3f9-60d476aa5680\") "
Nov 03 22:25:18.037897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.037869    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca" pod "87d21e97-52b7-48f2-b3f9-60d476aa5680" (UID: "87d21e97-52b7-48f2-b3f9-60d476aa5680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:25:18.039540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.039518    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz" pod "87d21e97-52b7-48f2-b3f9-60d476aa5680" (UID: "87d21e97-52b7-48f2-b3f9-60d476aa5680"). InnerVolumeSpecName "kube-api-access-hlnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:25:18.039619 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.039600    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca" pod "87d21e97-52b7-48f2-b3f9-60d476aa5680" (UID: "87d21e97-52b7-48f2-b3f9-60d476aa5680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:25:18.138768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.138687    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:18.138768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.138726    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnzz\" (UniqueName: \"kubernetes.io/projected/87d21e97-52b7-48f2-b3f9-60d476aa5680-kube-api-access-hlnzz\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:18.138768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.138742    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d21e97-52b7-48f2-b3f9-60d476aa5680-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:18.836712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.836648    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"
Nov 03 22:25:18.836712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.836682    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl" event={"ID":"87d21e97-52b7-48f2-b3f9-60d476aa5680","Type":"ContainerDied","Data":"0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855"}
Nov 03 22:25:18.836927 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:18.836734    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8f1918a528c3ceedb6ffade1c93e575bef2e45d327f8801f7fb2c94775f855"
Nov 03 22:25:19.260697 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:25:19.260665    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:25:22.740013 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:22.739981    5533 scope.go:122] "RemoveContainer" containerID="275fdcac126239f16dd3cad2e7fc9a7c747dabe5af965827543baf5eac648b5c"
Nov 03 22:25:27.715839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:27.715804    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:36.661456 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.661423    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"]
Nov 03 22:25:36.662055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.662021    5533 cpu_manager.go:414] "containerMap: removing container" podUID="87d21e97-52b7-48f2-b3f9-60d476aa5680" containerName="revoke-keys"
Nov 03 22:25:36.662055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.662042    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d21e97-52b7-48f2-b3f9-60d476aa5680" containerName="revoke-keys"
Nov 03 22:25:36.662186 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.662128    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d21e97-52b7-48f2-b3f9-60d476aa5680" containerName="revoke-keys"
Nov 03 22:25:36.666421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.666400    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.719711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.719683    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"]
Nov 03 22:25:36.723400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.723369    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.723547 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.723428    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2wq4\" (UniqueName: \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.723547 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.723466    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.763205 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.763171    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:25:36.763318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.763224    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:25:36.824764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.824740    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.824887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.824787    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2wq4\" (UniqueName: \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.824887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.824831    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.825561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.825539    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.827239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.827214    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.883294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.883271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2wq4\" (UniqueName: \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\") pod \"maas-api-revoke-sub-a4aef7d7-dpmzk\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:36.978758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:36.978726    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:37.345083 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:37.345058    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"]
Nov 03 22:25:37.347215 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:25:37.347187    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad3a441_fbd1_4c87_b071_8abd4e91fcb4.slice/crio-9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585 WatchSource:0}: Error finding container 9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585: Status 404 returned error can't find the container with id 9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585
Nov 03 22:25:37.407982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:37.407953    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:37.955453 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:37.955422    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk" event={"ID":"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4","Type":"ContainerStarted","Data":"7d17e7880e67f62ae8d39f6d9d366330b06df736e5fc9de3d399743730e49927"}
Nov 03 22:25:37.955861 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:37.955471    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk" event={"ID":"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4","Type":"ContainerStarted","Data":"9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585"}
Nov 03 22:25:38.065156 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:38.065103    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk" podStartSLOduration=2.065085254 podStartE2EDuration="2.065085254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:38.063893748 +0000 UTC m=+5839.695207237" watchObservedRunningTime="2026-11-03 22:25:38.065085254 +0000 UTC m=+5839.696398732"
Nov 03 22:25:38.963270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:38.963238    5533 generic.go:356] "Generic (PLEG): container finished" podID="7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" containerID="7d17e7880e67f62ae8d39f6d9d366330b06df736e5fc9de3d399743730e49927" exitCode=0
Nov 03 22:25:38.963713 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:38.963325    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk" event={"ID":"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4","Type":"ContainerDied","Data":"7d17e7880e67f62ae8d39f6d9d366330b06df736e5fc9de3d399743730e49927"}
Nov 03 22:25:40.097493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.097467    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:40.153284 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.153250    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\" (UniqueName: \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\") pod \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") "
Nov 03 22:25:40.153431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.153331    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\") pod \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") "
Nov 03 22:25:40.153431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.153391    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\") pod \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\" (UID: \"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4\") "
Nov 03 22:25:40.153868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.153835    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca" pod "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" (UID: "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:25:40.155533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.155497    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4" pod "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" (UID: "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4"). InnerVolumeSpecName "kube-api-access-z2wq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:25:40.156280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.156256    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca" pod "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" (UID: "7ad3a441-fbd1-4c87-b071-8abd4e91fcb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:25:40.255151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.255086    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:40.255151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.255121    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:40.255151 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.255136    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2wq4\" (UniqueName: \"kubernetes.io/projected/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4-kube-api-access-z2wq4\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:25:40.975293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.975263    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"
Nov 03 22:25:40.975459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.975311    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk" event={"ID":"7ad3a441-fbd1-4c87-b071-8abd4e91fcb4","Type":"ContainerDied","Data":"9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585"}
Nov 03 22:25:40.975459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:40.975386    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b88c6fc27672ca72d4d1d7115e34696325e8185282466c3d8db6ae191829585"
Nov 03 22:25:46.085575 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:46.085537    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:49.260579 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:25:49.260547    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:25:55.271233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:55.271208    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:25:57.840352 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.840291    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"]
Nov 03 22:25:57.841331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.841306    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" containerName="revoke-keys"
Nov 03 22:25:57.841331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.841330    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" containerName="revoke-keys"
Nov 03 22:25:57.841513 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.841490    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" containerName="revoke-keys"
Nov 03 22:25:57.845850 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.845827    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:57.902153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.902122    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"]
Nov 03 22:25:57.940714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.940257    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpvt\" (UniqueName: \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:57.940714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.940413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:57.940714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.940490    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:57.943779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.943756    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:25:57.943971 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:57.943956    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:25:58.041191 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.041159    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.041380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.041213    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.041380 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.041272    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpvt\" (UniqueName: \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.042101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.042069    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.047267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.047241    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.118145 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.118074    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpvt\" (UniqueName: \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\") pod \"maas-api-revoke-sub-79805ca9-nb498\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.176769 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.176736    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:25:58.652120 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:58.652093    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"]
Nov 03 22:25:58.654262 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:25:58.654231    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21629a0_f620_47d3_8755_b43604397ee0.slice/crio-454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103 WatchSource:0}: Error finding container 454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103: Status 404 returned error can't find the container with id 454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103
Nov 03 22:25:59.130239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:59.130166    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498" event={"ID":"c21629a0-f620-47d3-8755-b43604397ee0","Type":"ContainerStarted","Data":"46de56bf40246c818b16e824aeecb354e5042d24ac37e31bcbf2dff24bc06d47"}
Nov 03 22:25:59.130239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:59.130221    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498" event={"ID":"c21629a0-f620-47d3-8755-b43604397ee0","Type":"ContainerStarted","Data":"454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103"}
Nov 03 22:25:59.240412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:25:59.240357    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498" podStartSLOduration=2.240343849 podStartE2EDuration="2.240343849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:59.238290862 +0000 UTC m=+5860.869604341" watchObservedRunningTime="2026-11-03 22:25:59.240343849 +0000 UTC m=+5860.871657359"
Nov 03 22:26:00.137071 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:00.137041    5533 generic.go:356] "Generic (PLEG): container finished" podID="c21629a0-f620-47d3-8755-b43604397ee0" containerID="46de56bf40246c818b16e824aeecb354e5042d24ac37e31bcbf2dff24bc06d47" exitCode=0
Nov 03 22:26:00.137518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:00.137128    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498" event={"ID":"c21629a0-f620-47d3-8755-b43604397ee0","Type":"ContainerDied","Data":"46de56bf40246c818b16e824aeecb354e5042d24ac37e31bcbf2dff24bc06d47"}
Nov 03 22:26:01.279831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.279807    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:26:01.372631 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.372601    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\" (UniqueName: \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\") pod \"c21629a0-f620-47d3-8755-b43604397ee0\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") "
Nov 03 22:26:01.372795 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.372699    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\") pod \"c21629a0-f620-47d3-8755-b43604397ee0\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") "
Nov 03 22:26:01.372795 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.372776    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\") pod \"c21629a0-f620-47d3-8755-b43604397ee0\" (UID: \"c21629a0-f620-47d3-8755-b43604397ee0\") "
Nov 03 22:26:01.373077 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.373056    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca" pod "c21629a0-f620-47d3-8755-b43604397ee0" (UID: "c21629a0-f620-47d3-8755-b43604397ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:26:01.374771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.374747    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca" pod "c21629a0-f620-47d3-8755-b43604397ee0" (UID: "c21629a0-f620-47d3-8755-b43604397ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:26:01.374901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.374836    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt" pod "c21629a0-f620-47d3-8755-b43604397ee0" (UID: "c21629a0-f620-47d3-8755-b43604397ee0"). InnerVolumeSpecName "kube-api-access-mxpvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:26:01.473911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.473879    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:01.473911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.473908    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpvt\" (UniqueName: \"kubernetes.io/projected/c21629a0-f620-47d3-8755-b43604397ee0-kube-api-access-mxpvt\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:01.474101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:01.473924    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21629a0-f620-47d3-8755-b43604397ee0-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:02.149582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:02.149546    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"
Nov 03 22:26:02.149582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:02.149571    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498" event={"ID":"c21629a0-f620-47d3-8755-b43604397ee0","Type":"ContainerDied","Data":"454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103"}
Nov 03 22:26:02.149822 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:02.149621    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454dee7eaa63c353acf50d03975ef4ef00713af46de29f51b2908cbdc5a85103"
Nov 03 22:26:05.903136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:05.903100    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:26:15.241357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.241321    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"]
Nov 03 22:26:15.241846 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.241762    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c21629a0-f620-47d3-8755-b43604397ee0" containerName="revoke-keys"
Nov 03 22:26:15.241846 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.241774    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21629a0-f620-47d3-8755-b43604397ee0" containerName="revoke-keys"
Nov 03 22:26:15.241936 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.241852    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21629a0-f620-47d3-8755-b43604397ee0" containerName="revoke-keys"
Nov 03 22:26:15.245097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.245077    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.334857 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.334806    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"]
Nov 03 22:26:15.349283 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.349244    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:26:15.349808 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.349788    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:26:15.432350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.432306    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.432570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.432386    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.432570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.432453    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwmv\" (UniqueName: \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.533198 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.533103    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.533498 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.533475    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.533734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.533704    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwmv\" (UniqueName: \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.534058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.534003    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.536746 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.536718    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.592577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.592540    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwmv\" (UniqueName: \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\") pod \"maas-api-revoke-sub-0b7e8edb-tlv9x\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:15.880101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:15.880019    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:16.243650 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:16.243605    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"]
Nov 03 22:26:16.245362 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:26:16.245335    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e748ffe_fba6_40a2_bdda_1c53c1eb906d.slice/crio-85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c WatchSource:0}: Error finding container 85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c: Status 404 returned error can't find the container with id 85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c
Nov 03 22:26:17.256556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:17.256475    5533 generic.go:356] "Generic (PLEG): container finished" podID="7e748ffe-fba6-40a2-bdda-1c53c1eb906d" containerID="ef9230414849482bacbfa539530f9c1489fee37e6bdd2ff03e9d609b1e4ddcff" exitCode=0
Nov 03 22:26:17.256556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:17.256507    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x" event={"ID":"7e748ffe-fba6-40a2-bdda-1c53c1eb906d","Type":"ContainerDied","Data":"ef9230414849482bacbfa539530f9c1489fee37e6bdd2ff03e9d609b1e4ddcff"}
Nov 03 22:26:17.256989 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:17.256573    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x" event={"ID":"7e748ffe-fba6-40a2-bdda-1c53c1eb906d","Type":"ContainerStarted","Data":"85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c"}
Nov 03 22:26:18.391244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.391216    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:18.456592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.456564    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\" (UniqueName: \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\") pod \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") "
Nov 03 22:26:18.456755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.456672    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\") pod \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") "
Nov 03 22:26:18.456755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.456698    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\") pod \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\" (UID: \"7e748ffe-fba6-40a2-bdda-1c53c1eb906d\") "
Nov 03 22:26:18.457092 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.457067    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca" pod "7e748ffe-fba6-40a2-bdda-1c53c1eb906d" (UID: "7e748ffe-fba6-40a2-bdda-1c53c1eb906d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:26:18.458719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.458697    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca" pod "7e748ffe-fba6-40a2-bdda-1c53c1eb906d" (UID: "7e748ffe-fba6-40a2-bdda-1c53c1eb906d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:26:18.458832 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.458776    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv" pod "7e748ffe-fba6-40a2-bdda-1c53c1eb906d" (UID: "7e748ffe-fba6-40a2-bdda-1c53c1eb906d"). InnerVolumeSpecName "kube-api-access-hxwmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:26:18.557490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.557424    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwmv\" (UniqueName: \"kubernetes.io/projected/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-kube-api-access-hxwmv\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:18.557490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.557452    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:18.557490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.557462    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e748ffe-fba6-40a2-bdda-1c53c1eb906d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:18.674275 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:18.674248    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:26:19.259842 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:26:19.259810    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:26:19.270003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:19.269979    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x" event={"ID":"7e748ffe-fba6-40a2-bdda-1c53c1eb906d","Type":"ContainerDied","Data":"85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c"}
Nov 03 22:26:19.270155 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:19.270017    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"
Nov 03 22:26:19.270155 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:19.270022    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85135fd9e7519d0ec29031b3dfb1e1e133fc2c6edea7dedcaf46eb4868b5c53c"
Nov 03 22:26:22.761278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:22.761241    5533 scope.go:122] "RemoveContainer" containerID="39990a72d4857dc7159171a03ca2e8784c356482bc4ad6aa5580740ab581b6fe"
Nov 03 22:26:22.770406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:22.770381    5533 scope.go:122] "RemoveContainer" containerID="50dc9975ca791be3028513fbd1972a22c93e3ce5df21a456b93c411d98ec9866"
Nov 03 22:26:26.510034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.510000    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"]
Nov 03 22:26:26.510677 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.510656    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7e748ffe-fba6-40a2-bdda-1c53c1eb906d" containerName="revoke-keys"
Nov 03 22:26:26.510738 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.510680    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e748ffe-fba6-40a2-bdda-1c53c1eb906d" containerName="revoke-keys"
Nov 03 22:26:26.510844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.510831    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e748ffe-fba6-40a2-bdda-1c53c1eb906d" containerName="revoke-keys"
Nov 03 22:26:26.516115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.516096    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.567603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.567566    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"]
Nov 03 22:26:26.592795 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.592768    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:26:26.610625 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.610602    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:26:26.626022 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.625985    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.626126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.626053    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.626194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.626156    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9ck\" (UniqueName: \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.727093 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.727056    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9ck\" (UniqueName: \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.727235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.727152    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.727235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.727199    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.727819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.727793    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.729694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.729675    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.785902 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.785840    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9ck\" (UniqueName: \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\") pod \"maas-api-revoke-sub-3d4156f1-76fp2\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:26.832764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:26.832735    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:27.199461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:27.199426    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"]
Nov 03 22:26:27.200784 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:26:27.200757    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d838e64_cb2c_4cf0_99af_944c61c7bb81.slice/crio-7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b WatchSource:0}: Error finding container 7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b: Status 404 returned error can't find the container with id 7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b
Nov 03 22:26:27.279552 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:27.279521    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:26:27.313558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:27.313532    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2" event={"ID":"8d838e64-cb2c-4cf0-99af-944c61c7bb81","Type":"ContainerStarted","Data":"7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b"}
Nov 03 22:26:28.321592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:28.321513    5533 generic.go:356] "Generic (PLEG): container finished" podID="8d838e64-cb2c-4cf0-99af-944c61c7bb81" containerID="23f11a3956c67d717be9217c805285b2aa8c94e364da994be29019e79e18866c" exitCode=0
Nov 03 22:26:28.321974 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:28.321597    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2" event={"ID":"8d838e64-cb2c-4cf0-99af-944c61c7bb81","Type":"ContainerDied","Data":"23f11a3956c67d717be9217c805285b2aa8c94e364da994be29019e79e18866c"}
Nov 03 22:26:29.454270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.454247    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:29.551820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.551794    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\" (UniqueName: \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\") pod \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") "
Nov 03 22:26:29.551952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.551877    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\") pod \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") "
Nov 03 22:26:29.551952 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.551902    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\") pod \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\" (UID: \"8d838e64-cb2c-4cf0-99af-944c61c7bb81\") "
Nov 03 22:26:29.552332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.552308    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca" pod "8d838e64-cb2c-4cf0-99af-944c61c7bb81" (UID: "8d838e64-cb2c-4cf0-99af-944c61c7bb81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:26:29.553811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.553786    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck" pod "8d838e64-cb2c-4cf0-99af-944c61c7bb81" (UID: "8d838e64-cb2c-4cf0-99af-944c61c7bb81"). InnerVolumeSpecName "kube-api-access-vw9ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:26:29.553906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.553869    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca" pod "8d838e64-cb2c-4cf0-99af-944c61c7bb81" (UID: "8d838e64-cb2c-4cf0-99af-944c61c7bb81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:26:29.652880 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.652808    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9ck\" (UniqueName: \"kubernetes.io/projected/8d838e64-cb2c-4cf0-99af-944c61c7bb81-kube-api-access-vw9ck\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:29.652880 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.652838    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:29.652880 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:29.652849    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d838e64-cb2c-4cf0-99af-944c61c7bb81-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:30.333371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:30.333340    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"
Nov 03 22:26:30.333565 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:30.333386    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2" event={"ID":"8d838e64-cb2c-4cf0-99af-944c61c7bb81","Type":"ContainerDied","Data":"7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b"}
Nov 03 22:26:30.333565 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:30.333435    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bffe33358c413caa9b9074da36233499e6aade9bc019667db0a06f5fcc1fc2b"
Nov 03 22:26:39.397502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:39.397462    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:26:48.104229 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.104194    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"]
Nov 03 22:26:48.150984 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.150952    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-76fp2"]
Nov 03 22:26:48.230407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.230375    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"]
Nov 03 22:26:48.230777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.230763    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8d838e64-cb2c-4cf0-99af-944c61c7bb81" containerName="revoke-keys"
Nov 03 22:26:48.230838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.230778    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d838e64-cb2c-4cf0-99af-944c61c7bb81" containerName="revoke-keys"
Nov 03 22:26:48.230882 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.230851    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d838e64-cb2c-4cf0-99af-944c61c7bb81" containerName="revoke-keys"
Nov 03 22:26:48.234117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.234097    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.287665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.287615    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:26:48.287805 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.287688    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:26:48.289252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.289222    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"]
Nov 03 22:26:48.343357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.343334    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dpw\" (UniqueName: \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.343478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.343403    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.343523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.343481    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.444886 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.444859    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.444993 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.444891    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.445086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.445074    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dpw\" (UniqueName: \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.445570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.445549    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.447415 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.447393    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.492199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.492167    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d838e64-cb2c-4cf0-99af-944c61c7bb81" path="/var/lib/kubelet/pods/8d838e64-cb2c-4cf0-99af-944c61c7bb81/volumes"
Nov 03 22:26:48.504523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.504503    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dpw\" (UniqueName: \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\") pod \"maas-api-revoke-sub-3d4156f1-j4l42\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.546501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.546480    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:48.919690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:48.919662    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"]
Nov 03 22:26:49.261359 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:26:49.261329    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:26:49.283153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:49.283128    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:26:49.458614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:49.458587    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42" event={"ID":"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7","Type":"ContainerStarted","Data":"8205d0b67b863e158351139a00d9a40785b162e41381e4c4f46c107df4a19b52"}
Nov 03 22:26:49.458757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:49.458658    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42" event={"ID":"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7","Type":"ContainerStarted","Data":"57348916307caa8f7819befe62546f81c0e044b2c020d41550281d3e2f9bd9e4"}
Nov 03 22:26:49.574910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:49.574837    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42" podStartSLOduration=1.574824706 podStartE2EDuration="1.574824706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:49.572528082 +0000 UTC m=+5911.203841559" watchObservedRunningTime="2026-11-03 22:26:49.574824706 +0000 UTC m=+5911.206138182"
Nov 03 22:26:50.465863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:50.465829    5533 generic.go:356] "Generic (PLEG): container finished" podID="2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" containerID="8205d0b67b863e158351139a00d9a40785b162e41381e4c4f46c107df4a19b52" exitCode=0
Nov 03 22:26:50.466280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:50.465918    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42" event={"ID":"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7","Type":"ContainerDied","Data":"8205d0b67b863e158351139a00d9a40785b162e41381e4c4f46c107df4a19b52"}
Nov 03 22:26:51.600690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.600662    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:51.674026 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.673999    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\") pod \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") "
Nov 03 22:26:51.674161 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.674141    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\" (UniqueName: \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\") pod \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") "
Nov 03 22:26:51.674227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.674187    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\") pod \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\" (UID: \"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7\") "
Nov 03 22:26:51.674582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.674561    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca" pod "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" (UID: "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:26:51.676059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.676033    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca" pod "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" (UID: "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:26:51.676214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.676189    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw" pod "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" (UID: "2f7d0e2f-3ca1-4933-860f-98e49a16d2e7"). InnerVolumeSpecName "kube-api-access-q2dpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:26:51.774899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.774840    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dpw\" (UniqueName: \"kubernetes.io/projected/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-kube-api-access-q2dpw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:51.774899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.774867    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:51.774899 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:51.774878    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:26:52.480499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:52.480464    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"
Nov 03 22:26:52.480703 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:52.480494    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42" event={"ID":"2f7d0e2f-3ca1-4933-860f-98e49a16d2e7","Type":"ContainerDied","Data":"57348916307caa8f7819befe62546f81c0e044b2c020d41550281d3e2f9bd9e4"}
Nov 03 22:26:52.480703 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:26:52.480565    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57348916307caa8f7819befe62546f81c0e044b2c020d41550281d3e2f9bd9e4"
Nov 03 22:27:02.901082 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:02.901047    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:27:13.198914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:13.198882    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:27:19.259933 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:27:19.259898    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:27:23.102972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:23.102931    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:27:23.103957 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:23.103627    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:27:23.123271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:23.123163    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:27:23.123271 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:23.123214    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:27:25.020558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:25.020514    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:27:33.403085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.403044    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"]
Nov 03 22:27:33.403496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.403413    5533 cpu_manager.go:414] "containerMap: removing container" podUID="2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" containerName="revoke-keys"
Nov 03 22:27:33.403496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.403424    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" containerName="revoke-keys"
Nov 03 22:27:33.403577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.403504    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" containerName="revoke-keys"
Nov 03 22:27:33.408260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.408235    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.429295 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.429269    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.429405 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.429311    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.429405 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.429336    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpj8z\" (UniqueName: \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.461176 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.461151    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"]
Nov 03 22:27:33.504401 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.504379    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:27:33.504538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.504447    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:27:33.529915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.529885    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.530046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.529947    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.530046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.529996    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpj8z\" (UniqueName: \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.530590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.530571    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.532290 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.532271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.587538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.587517    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpj8z\" (UniqueName: \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\") pod \"maas-api-revoke-sub-e2fc9c5b-g2cgk\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:33.721328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:33.721307    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:34.085985 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:34.085950    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"]
Nov 03 22:27:34.093794 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:27:34.093764    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0751a82_bc21_4014_8fa9_0790099f7493.slice/crio-3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580 WatchSource:0}: Error finding container 3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580: Status 404 returned error can't find the container with id 3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580
Nov 03 22:27:34.717578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:34.717538    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk" event={"ID":"f0751a82-bc21-4014-8fa9-0790099f7493","Type":"ContainerStarted","Data":"c07d1ecfa4378dcbf108aa3ea58c1a3724b1ed79c380bf824d2e6065218a25a0"}
Nov 03 22:27:34.718096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:34.717588    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk" event={"ID":"f0751a82-bc21-4014-8fa9-0790099f7493","Type":"ContainerStarted","Data":"3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580"}
Nov 03 22:27:34.827420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:34.827366    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk" podStartSLOduration=1.8273507150000001 podStartE2EDuration="1.827350715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:34.827139276 +0000 UTC m=+5956.458452755" watchObservedRunningTime="2026-11-03 22:27:34.827350715 +0000 UTC m=+5956.458664193"
Nov 03 22:27:35.724527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:35.724479    5533 generic.go:356] "Generic (PLEG): container finished" podID="f0751a82-bc21-4014-8fa9-0790099f7493" containerID="c07d1ecfa4378dcbf108aa3ea58c1a3724b1ed79c380bf824d2e6065218a25a0" exitCode=0
Nov 03 22:27:35.724913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:35.724563    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk" event={"ID":"f0751a82-bc21-4014-8fa9-0790099f7493","Type":"ContainerDied","Data":"c07d1ecfa4378dcbf108aa3ea58c1a3724b1ed79c380bf824d2e6065218a25a0"}
Nov 03 22:27:36.857258 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.857236    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:36.960506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.960475    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\") pod \"f0751a82-bc21-4014-8fa9-0790099f7493\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") "
Nov 03 22:27:36.960664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.960523    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\" (UniqueName: \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\") pod \"f0751a82-bc21-4014-8fa9-0790099f7493\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") "
Nov 03 22:27:36.960748 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.960670    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\") pod \"f0751a82-bc21-4014-8fa9-0790099f7493\" (UID: \"f0751a82-bc21-4014-8fa9-0790099f7493\") "
Nov 03 22:27:36.960894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.960868    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca" pod "f0751a82-bc21-4014-8fa9-0790099f7493" (UID: "f0751a82-bc21-4014-8fa9-0790099f7493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:27:36.962524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.962500    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z" pod "f0751a82-bc21-4014-8fa9-0790099f7493" (UID: "f0751a82-bc21-4014-8fa9-0790099f7493"). InnerVolumeSpecName "kube-api-access-vpj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:27:36.962622 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:36.962569    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca" pod "f0751a82-bc21-4014-8fa9-0790099f7493" (UID: "f0751a82-bc21-4014-8fa9-0790099f7493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:27:37.062073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.062001    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:37.062073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.062027    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0751a82-bc21-4014-8fa9-0790099f7493-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:37.062073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.062038    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpj8z\" (UniqueName: \"kubernetes.io/projected/f0751a82-bc21-4014-8fa9-0790099f7493-kube-api-access-vpj8z\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:37.736156 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.736110    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk" event={"ID":"f0751a82-bc21-4014-8fa9-0790099f7493","Type":"ContainerDied","Data":"3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580"}
Nov 03 22:27:37.736339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.736172    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7bf4e5cca2f07f67f6a05a4588a85903010dafb1bc0dd0decf1f5dae467580"
Nov 03 22:27:37.736339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:37.736130    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"
Nov 03 22:27:44.218951 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.218917    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"]
Nov 03 22:27:44.219370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.219281    5533 cpu_manager.go:414] "containerMap: removing container" podUID="f0751a82-bc21-4014-8fa9-0790099f7493" containerName="revoke-keys"
Nov 03 22:27:44.219370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.219292    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0751a82-bc21-4014-8fa9-0790099f7493" containerName="revoke-keys"
Nov 03 22:27:44.219466 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.219374    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0751a82-bc21-4014-8fa9-0790099f7493" containerName="revoke-keys"
Nov 03 22:27:44.222327 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.222309    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.274184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.274158    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:27:44.319391 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.319361    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5d5d\" (UniqueName: \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.319546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.319395    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.319546 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.319523    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.320915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.320892    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:27:44.324019 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.323998    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"]
Nov 03 22:27:44.419960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.419929    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.420095 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.419978    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5d5d\" (UniqueName: \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.420095 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.419996    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.420644 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.420610    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.422215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.422191    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.477367 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.477300    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5d5d\" (UniqueName: \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\") pod \"maas-api-revoke-sub-5a29efb3-h2czg\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.537782 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.537753    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:44.903136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:44.903110    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"]
Nov 03 22:27:44.904889 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:27:44.904857    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695a39dc_648b_43f9_a4ce_2c4e22c149f6.slice/crio-1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851 WatchSource:0}: Error finding container 1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851: Status 404 returned error can't find the container with id 1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851
Nov 03 22:27:45.781288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:45.781250    5533 generic.go:356] "Generic (PLEG): container finished" podID="695a39dc-648b-43f9-a4ce-2c4e22c149f6" containerID="e70002df07c0b5a34ba4ff266e8e9e7b2aabddfdbbebb911477c2ed76d8ecb62" exitCode=0
Nov 03 22:27:45.781714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:45.781315    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg" event={"ID":"695a39dc-648b-43f9-a4ce-2c4e22c149f6","Type":"ContainerDied","Data":"e70002df07c0b5a34ba4ff266e8e9e7b2aabddfdbbebb911477c2ed76d8ecb62"}
Nov 03 22:27:45.781714 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:45.781369    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg" event={"ID":"695a39dc-648b-43f9-a4ce-2c4e22c149f6","Type":"ContainerStarted","Data":"1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851"}
Nov 03 22:27:46.295106 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:46.295064    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:27:46.932697 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:46.932674    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:47.046400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.045887    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\") pod \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") "
Nov 03 22:27:47.046400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.045998    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\" (UniqueName: \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\") pod \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") "
Nov 03 22:27:47.046708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.046612    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca" pod "695a39dc-648b-43f9-a4ce-2c4e22c149f6" (UID: "695a39dc-648b-43f9-a4ce-2c4e22c149f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:27:47.046708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.046685    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\") pod \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\" (UID: \"695a39dc-648b-43f9-a4ce-2c4e22c149f6\") "
Nov 03 22:27:47.047035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.046998    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:47.050786 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.050747    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d" pod "695a39dc-648b-43f9-a4ce-2c4e22c149f6" (UID: "695a39dc-648b-43f9-a4ce-2c4e22c149f6"). InnerVolumeSpecName "kube-api-access-n5d5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:27:47.057949 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.057739    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca" pod "695a39dc-648b-43f9-a4ce-2c4e22c149f6" (UID: "695a39dc-648b-43f9-a4ce-2c4e22c149f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:27:47.148581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.148533    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5d5d\" (UniqueName: \"kubernetes.io/projected/695a39dc-648b-43f9-a4ce-2c4e22c149f6-kube-api-access-n5d5d\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:47.148581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.148576    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695a39dc-648b-43f9-a4ce-2c4e22c149f6-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:27:47.828926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.828887    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg" event={"ID":"695a39dc-648b-43f9-a4ce-2c4e22c149f6","Type":"ContainerDied","Data":"1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851"}
Nov 03 22:27:47.829174 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.829153    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b63e3539cbaf291e1da3399a7438be36dece257b7fdd9a6c0220714d9b37851"
Nov 03 22:27:47.829389 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:47.829373    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"
Nov 03 22:27:49.047790 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:49.047748    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"]
Nov 03 22:27:49.126381 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:49.126339    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-krktf"]
Nov 03 22:27:49.272097 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:27:49.272026    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:27:50.493526 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:50.493490    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01a5fb1-9185-4568-9a6f-5a3f49341a63" path="/var/lib/kubelet/pods/b01a5fb1-9185-4568-9a6f-5a3f49341a63/volumes"
Nov 03 22:27:56.991049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:27:56.991013    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:28:00.959573 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:00.959540    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"]
Nov 03 22:28:01.006373 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.006344    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j4l42"]
Nov 03 22:28:01.085980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.085947    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"]
Nov 03 22:28:01.086403 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.086387    5533 cpu_manager.go:414] "containerMap: removing container" podUID="695a39dc-648b-43f9-a4ce-2c4e22c149f6" containerName="revoke-keys"
Nov 03 22:28:01.086486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.086403    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695a39dc-648b-43f9-a4ce-2c4e22c149f6" containerName="revoke-keys"
Nov 03 22:28:01.086486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.086475    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="695a39dc-648b-43f9-a4ce-2c4e22c149f6" containerName="revoke-keys"
Nov 03 22:28:01.090763 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.090738    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.121387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.121361    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.121541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.121397    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjls\" (UniqueName: \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.121541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.121494    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.141246 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.141214    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:28:01.141425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.141404    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:28:01.144189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.144161    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"]
Nov 03 22:28:01.222990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.222929    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.223128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.223023    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.223128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.223048    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjls\" (UniqueName: \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.223554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.223531    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.225244 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.225223    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.280716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.280689    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjls\" (UniqueName: \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\") pod \"maas-api-revoke-sub-3d4156f1-5x82x\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.405125 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.405100    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:01.771887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.771856    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"]
Nov 03 22:28:01.773263 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:28:01.773229    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df09764_2383_4b14_a7e3_7353795714be.slice/crio-f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38 WatchSource:0}: Error finding container f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38: Status 404 returned error can't find the container with id f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38
Nov 03 22:28:01.910872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:01.910841    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x" event={"ID":"0df09764-2383-4b14-a7e3-7353795714be","Type":"ContainerStarted","Data":"f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38"}
Nov 03 22:28:02.492343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:02.492316    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7d0e2f-3ca1-4933-860f-98e49a16d2e7" path="/var/lib/kubelet/pods/2f7d0e2f-3ca1-4933-860f-98e49a16d2e7/volumes"
Nov 03 22:28:02.918024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:02.917955    5533 generic.go:356] "Generic (PLEG): container finished" podID="0df09764-2383-4b14-a7e3-7353795714be" containerID="4a393f718d26f0e9297e1dea2449a2960fbf6cf8019317ffedb8aaca7710a199" exitCode=0
Nov 03 22:28:02.918160 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:02.918040    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x" event={"ID":"0df09764-2383-4b14-a7e3-7353795714be","Type":"ContainerDied","Data":"4a393f718d26f0e9297e1dea2449a2960fbf6cf8019317ffedb8aaca7710a199"}
Nov 03 22:28:04.054168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.054145    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:04.148260 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.148236    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\") pod \"0df09764-2383-4b14-a7e3-7353795714be\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") "
Nov 03 22:28:04.148394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.148281    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\") pod \"0df09764-2383-4b14-a7e3-7353795714be\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") "
Nov 03 22:28:04.148394 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.148350    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\" (UniqueName: \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\") pod \"0df09764-2383-4b14-a7e3-7353795714be\" (UID: \"0df09764-2383-4b14-a7e3-7353795714be\") "
Nov 03 22:28:04.148573 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.148549    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca" pod "0df09764-2383-4b14-a7e3-7353795714be" (UID: "0df09764-2383-4b14-a7e3-7353795714be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:28:04.148675 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.148632    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df09764-2383-4b14-a7e3-7353795714be-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:28:04.150347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.150324    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls" pod "0df09764-2383-4b14-a7e3-7353795714be" (UID: "0df09764-2383-4b14-a7e3-7353795714be"). InnerVolumeSpecName "kube-api-access-9tjls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:28:04.150423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.150370    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca" pod "0df09764-2383-4b14-a7e3-7353795714be" (UID: "0df09764-2383-4b14-a7e3-7353795714be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:28:04.249978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.249902    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df09764-2383-4b14-a7e3-7353795714be-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:28:04.249978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.249934    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjls\" (UniqueName: \"kubernetes.io/projected/0df09764-2383-4b14-a7e3-7353795714be-kube-api-access-9tjls\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:28:04.930294 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.930266    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"
Nov 03 22:28:04.930469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.930303    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x" event={"ID":"0df09764-2383-4b14-a7e3-7353795714be","Type":"ContainerDied","Data":"f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38"}
Nov 03 22:28:04.930469 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:04.930355    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19f797700189a7275e203c4452e1db46b1bf556314b6377769992f36c29dc38"
Nov 03 22:28:15.870859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:15.870823    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:28:19.261618 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:28:19.261583    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:28:19.565317 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:19.565243    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:28:19.576119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:19.576098    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:28:19.611746 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:19.611722    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:28:19.622421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:19.622402    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:28:20.045152 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:20.045109    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"]
Nov 03 22:28:20.091757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:20.091719    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dx897"]
Nov 03 22:28:20.492249 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:20.492217    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ded5521-27a6-4410-a172-61baf7b40e5e" path="/var/lib/kubelet/pods/2ded5521-27a6-4410-a172-61baf7b40e5e/volumes"
Nov 03 22:28:27.437990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.437948    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:28:27.438515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.438493    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0df09764-2383-4b14-a7e3-7353795714be" containerName="revoke-keys"
Nov 03 22:28:27.438576 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.438519    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df09764-2383-4b14-a7e3-7353795714be" containerName="revoke-keys"
Nov 03 22:28:27.438625 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.438606    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df09764-2383-4b14-a7e3-7353795714be" containerName="revoke-keys"
Nov 03 22:28:27.442972 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.442948    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.497109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.497079    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:28:27.501684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.501652    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.501829 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.501776    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r6l\" (UniqueName: \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.540937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.540914    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-51b683\""
Nov 03 22:28:27.540937 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.540926    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e1e41a32-dockercfg-6gdx2\""
Nov 03 22:28:27.598926 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.598899    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:28:27.602416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.602396    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.602580 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.602554    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.602674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.602617    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r6l\" (UniqueName: \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.603318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.603295    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.662388 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.662362    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:28:27.670169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.670146    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r6l\" (UniqueName: \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\") pod \"payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.703855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.703803    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.703855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.703836    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfs9\" (UniqueName: \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.756665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.756622    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:27.805371 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.805325    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.805554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.805405    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfs9\" (UniqueName: \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.810877 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.806796    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.865444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.865413    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfs9\" (UniqueName: \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\") pod \"payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:27.915653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:27.915615    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:28.128720 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:28.128694    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:28:28.129974 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:28:28.129944    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7c2061_2499_496f_ad9a_118aeff161ea.slice/crio-e6f9b71ab5179f6a22bdc8ae48f182dcafd52240664be145d663ec0a72fe86ad WatchSource:0}: Error finding container e6f9b71ab5179f6a22bdc8ae48f182dcafd52240664be145d663ec0a72fe86ad: Status 404 returned error can't find the container with id e6f9b71ab5179f6a22bdc8ae48f182dcafd52240664be145d663ec0a72fe86ad
Nov 03 22:28:28.282797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:28.282772    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:28:28.284097 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:28:28.284078    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b26ebae_b2da_49d5_b49b_7f3119c64c8d.slice/crio-1a268ac351e2ab4ada13310e7662690c51568086b23404ef53f57d4a524755b9 WatchSource:0}: Error finding container 1a268ac351e2ab4ada13310e7662690c51568086b23404ef53f57d4a524755b9: Status 404 returned error can't find the container with id 1a268ac351e2ab4ada13310e7662690c51568086b23404ef53f57d4a524755b9
Nov 03 22:28:29.103430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.103387    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" event={"ID":"4f7c2061-2499-496f-ad9a-118aeff161ea","Type":"ContainerStarted","Data":"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"}
Nov 03 22:28:29.103929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.103462    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:29.103929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.103484    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" event={"ID":"4f7c2061-2499-496f-ad9a-118aeff161ea","Type":"ContainerStarted","Data":"e6f9b71ab5179f6a22bdc8ae48f182dcafd52240664be145d663ec0a72fe86ad"}
Nov 03 22:28:29.104995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.104972    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" event={"ID":"4b26ebae-b2da-49d5-b49b-7f3119c64c8d","Type":"ContainerStarted","Data":"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"}
Nov 03 22:28:29.105109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.105011    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" event={"ID":"4b26ebae-b2da-49d5-b49b-7f3119c64c8d","Type":"ContainerStarted","Data":"1a268ac351e2ab4ada13310e7662690c51568086b23404ef53f57d4a524755b9"}
Nov 03 22:28:29.105109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.105087    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:29.214648 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.214597    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" podStartSLOduration=2.214583375 podStartE2EDuration="2.214583375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:29.212783968 +0000 UTC m=+6010.844097444" watchObservedRunningTime="2026-11-03 22:28:29.214583375 +0000 UTC m=+6010.845896914"
Nov 03 22:28:29.335766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.335702    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" podStartSLOduration=2.335682714 podStartE2EDuration="2.335682714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:29.330999557 +0000 UTC m=+6010.962313034" watchObservedRunningTime="2026-11-03 22:28:29.335682714 +0000 UTC m=+6010.966996188"
Nov 03 22:28:29.607938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:29.607898    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:28:40.114572 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:40.114540    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:28:40.115116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:40.114605    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:28:41.488432 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:41.488396    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:28:45.046002 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:45.045964    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"]
Nov 03 22:28:45.092825 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:45.092790    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fx2pz"]
Nov 03 22:28:46.492321 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:46.492289    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b4556b-585f-4085-bcd1-8787056f1f04" path="/var/lib/kubelet/pods/55b4556b-585f-4085-bcd1-8787056f1f04/volumes"
Nov 03 22:28:49.260424 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:28:49.260388    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:28:56.768104 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:56.768059    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:28:58.050828 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:58.050790    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"]
Nov 03 22:28:58.097448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:58.097409    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2bnh4"]
Nov 03 22:28:58.491931 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:28:58.491892    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363093ed-6a0e-43ee-b7ec-fd162bc3abba" path="/var/lib/kubelet/pods/363093ed-6a0e-43ee-b7ec-fd162bc3abba/volumes"
Nov 03 22:29:19.260731 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:29:19.260625    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:29:22.318199 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:22.318165    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:29:22.840864 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:22.840838    5533 scope.go:122] "RemoveContainer" containerID="1963a2d2897bacff3bf5ea5494e57f4073b7f621dbbe41cf9d0214ac3775c83e"
Nov 03 22:29:22.849517 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:22.849494    5533 scope.go:122] "RemoveContainer" containerID="d836698ddd27a7490f3800a286b403f9a8648861ff4008ae7402a20ee1c306ed"
Nov 03 22:29:33.046411 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:33.046373    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"]
Nov 03 22:29:33.093170 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:33.093143    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zqkfn"]
Nov 03 22:29:34.492050 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:34.492012    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bbe15e-9623-408a-a877-20d54c91c8f3" path="/var/lib/kubelet/pods/83bbe15e-9623-408a-a877-20d54c91c8f3/volumes"
Nov 03 22:29:36.389418 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.389384    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"]
Nov 03 22:29:36.393437 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.393411    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.442448 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.442421    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:29:36.489122 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.489097    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:29:36.491614 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.491591    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"]
Nov 03 22:29:36.499123 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.499093    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m875b\" (UniqueName: \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.499236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.499194    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.499236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.499228    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.600541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.600514    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.600667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.600562    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.600727 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.600703    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m875b\" (UniqueName: \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.601203 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.601172    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.602941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.602914    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.658528 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.658476    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m875b\" (UniqueName: \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\") pod \"maas-api-revoke-sub-c288d3aa-sl94h\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:36.707278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:36.707258    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:37.072318 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:37.072181    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"]
Nov 03 22:29:37.074916 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:29:37.074880    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fa7022_0027_4433_864f_fded610f151c.slice/crio-abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c WatchSource:0}: Error finding container abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c: Status 404 returned error can't find the container with id abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c
Nov 03 22:29:37.505341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:37.505303    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h" event={"ID":"57fa7022-0027-4433-864f-fded610f151c","Type":"ContainerStarted","Data":"bbae25d878e3ba9e868daed3f8da53c986ced156e0f53e1f569c06df98e799a0"}
Nov 03 22:29:37.505708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:37.505359    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h" event={"ID":"57fa7022-0027-4433-864f-fded610f151c","Type":"ContainerStarted","Data":"abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c"}
Nov 03 22:29:37.616168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:37.616089    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h" podStartSLOduration=1.616075253 podStartE2EDuration="1.616075253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:37.613974568 +0000 UTC m=+6079.245288046" watchObservedRunningTime="2026-11-03 22:29:37.616075253 +0000 UTC m=+6079.247388729"
Nov 03 22:29:38.511866 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:38.511837    5533 generic.go:356] "Generic (PLEG): container finished" podID="57fa7022-0027-4433-864f-fded610f151c" containerID="bbae25d878e3ba9e868daed3f8da53c986ced156e0f53e1f569c06df98e799a0" exitCode=0
Nov 03 22:29:38.512241 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:38.511919    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h" event={"ID":"57fa7022-0027-4433-864f-fded610f151c","Type":"ContainerDied","Data":"bbae25d878e3ba9e868daed3f8da53c986ced156e0f53e1f569c06df98e799a0"}
Nov 03 22:29:39.647181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.647158    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:39.727451 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.727425    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\") pod \"57fa7022-0027-4433-864f-fded610f151c\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") "
Nov 03 22:29:39.727587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.727475    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\" (UniqueName: \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\") pod \"57fa7022-0027-4433-864f-fded610f151c\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") "
Nov 03 22:29:39.727587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.727513    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\") pod \"57fa7022-0027-4433-864f-fded610f151c\" (UID: \"57fa7022-0027-4433-864f-fded610f151c\") "
Nov 03 22:29:39.727942 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.727920    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca" pod "57fa7022-0027-4433-864f-fded610f151c" (UID: "57fa7022-0027-4433-864f-fded610f151c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:29:39.729564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.729541    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca" pod "57fa7022-0027-4433-864f-fded610f151c" (UID: "57fa7022-0027-4433-864f-fded610f151c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:29:39.729701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.729610    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b" pod "57fa7022-0027-4433-864f-fded610f151c" (UID: "57fa7022-0027-4433-864f-fded610f151c"). InnerVolumeSpecName "kube-api-access-m875b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:29:39.828740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.828681    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fa7022-0027-4433-864f-fded610f151c-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:29:39.828740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.828705    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fa7022-0027-4433-864f-fded610f151c-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:29:39.828740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:39.828716    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m875b\" (UniqueName: \"kubernetes.io/projected/57fa7022-0027-4433-864f-fded610f151c-kube-api-access-m875b\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:29:40.524440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:40.524413    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"
Nov 03 22:29:40.524578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:40.524439    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h" event={"ID":"57fa7022-0027-4433-864f-fded610f151c","Type":"ContainerDied","Data":"abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c"}
Nov 03 22:29:40.524578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:40.524488    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abeb9daf1f70a13843e20d55db05421b6a6b11fa068fb0263c1a37e96043a74c"
Nov 03 22:29:43.049024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:43.048987    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"]
Nov 03 22:29:43.095774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:43.095741    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ccmnz"]
Nov 03 22:29:44.491550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:44.491515    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe29570-6b24-4710-9581-792ec762431d" path="/var/lib/kubelet/pods/2fe29570-6b24-4710-9581-792ec762431d/volumes"
Nov 03 22:29:49.260509 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:29:49.260474    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:29:51.311391 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:51.311352    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:29:55.170540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:55.170506    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:29:55.278597 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:29:55.278569    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:30:00.157296 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.157260    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:00.157708 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.157693    5533 cpu_manager.go:414] "containerMap: removing container" podUID="57fa7022-0027-4433-864f-fded610f151c" containerName="revoke-keys"
Nov 03 22:30:00.157770 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.157711    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fa7022-0027-4433-864f-fded610f151c" containerName="revoke-keys"
Nov 03 22:30:00.157818 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.157808    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fa7022-0027-4433-864f-fded610f151c" containerName="revoke-keys"
Nov 03 22:30:00.162518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.162497    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.170905 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.170861    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing" containerID="cri-o://e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a" gracePeriod=25
Nov 03 22:30:00.258860 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.258833    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:30:00.261162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.261141    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:00.279034 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.279000    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing" containerID="cri-o://5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1" gracePeriod=25
Nov 03 22:30:00.303486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.303456    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8td8\" (UniqueName: \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.303603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.303505    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.404121 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.404094    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.404227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.404172    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8td8\" (UniqueName: \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.404745 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.404725    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.462044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.462017    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8td8\" (UniqueName: \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\") pod \"maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.475726 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.475707    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:00.841288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.841262    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:00.842662 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:30:00.842601    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c49001a_4a42_4b7b_9770_58510f222e4b.slice/crio-019485846fe6795b161f051e28b5bb4346efe0512c03594779b76fa6cf9d8d00 WatchSource:0}: Error finding container 019485846fe6795b161f051e28b5bb4346efe0512c03594779b76fa6cf9d8d00: Status 404 returned error can't find the container with id 019485846fe6795b161f051e28b5bb4346efe0512c03594779b76fa6cf9d8d00
Nov 03 22:30:00.929749 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:00.929722    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:01.649029 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.648995    5533 generic.go:356] "Generic (PLEG): container finished" podID="7c49001a-4a42-4b7b-9770-58510f222e4b" containerID="44b7c2eb1a0024b9948912aa01394e69f415771e44e3c6a0863f4742f89d834f" exitCode=6
Nov 03 22:30:01.649401 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.649080    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj" event={"ID":"7c49001a-4a42-4b7b-9770-58510f222e4b","Type":"ContainerDied","Data":"44b7c2eb1a0024b9948912aa01394e69f415771e44e3c6a0863f4742f89d834f"}
Nov 03 22:30:01.649401 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.649143    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj" event={"ID":"7c49001a-4a42-4b7b-9770-58510f222e4b","Type":"ContainerStarted","Data":"019485846fe6795b161f051e28b5bb4346efe0512c03594779b76fa6cf9d8d00"}
Nov 03 22:30:01.794000 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.793976    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:01.918006 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.917915    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\" (UniqueName: \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\") pod \"7c49001a-4a42-4b7b-9770-58510f222e4b\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") "
Nov 03 22:30:01.918006 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.917992    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\") pod \"7c49001a-4a42-4b7b-9770-58510f222e4b\" (UID: \"7c49001a-4a42-4b7b-9770-58510f222e4b\") "
Nov 03 22:30:01.918576 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.918543    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca" pod "7c49001a-4a42-4b7b-9770-58510f222e4b" (UID: "7c49001a-4a42-4b7b-9770-58510f222e4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:01.920055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:01.920027    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8" pod "7c49001a-4a42-4b7b-9770-58510f222e4b" (UID: "7c49001a-4a42-4b7b-9770-58510f222e4b"). InnerVolumeSpecName "kube-api-access-x8td8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:02.019409 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.019371    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8td8\" (UniqueName: \"kubernetes.io/projected/7c49001a-4a42-4b7b-9770-58510f222e4b-kube-api-access-x8td8\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:02.019409 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.019403    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c49001a-4a42-4b7b-9770-58510f222e4b-maas-api-service-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:02.656328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.656293    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"
Nov 03 22:30:02.656819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.656344    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj" event={"ID":"7c49001a-4a42-4b7b-9770-58510f222e4b","Type":"ContainerDied","Data":"019485846fe6795b161f051e28b5bb4346efe0512c03594779b76fa6cf9d8d00"}
Nov 03 22:30:02.656819 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.656447    5533 scope.go:122] "RemoveContainer" containerID="44b7c2eb1a0024b9948912aa01394e69f415771e44e3c6a0863f4742f89d834f"
Nov 03 22:30:02.810296 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.810227    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:02.856864 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:02.856839    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-51b683-298957588qqj"]
Nov 03 22:30:04.492460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:04.492423    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c49001a-4a42-4b7b-9770-58510f222e4b" path="/var/lib/kubelet/pods/7c49001a-4a42-4b7b-9770-58510f222e4b/volumes"
Nov 03 22:30:06.416630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:06.416587    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:30:10.113362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:10.113327    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:30:10.113775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:10.113322    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:30:18.045705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:18.045672    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"]
Nov 03 22:30:18.092146 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:18.092114    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6rsl"]
Nov 03 22:30:18.492527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:18.492501    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d21e97-52b7-48f2-b3f9-60d476aa5680" path="/var/lib/kubelet/pods/87d21e97-52b7-48f2-b3f9-60d476aa5680/volumes"
Nov 03 22:30:19.260229 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:30:19.260192    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:30:20.113826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.113778    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:30:20.113826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.113805    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:30:20.417338 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.417307    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:30:20.470044 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.470014    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\") pod \"4f7c2061-2499-496f-ad9a-118aeff161ea\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") "
Nov 03 22:30:20.470196 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.470146    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\" (UniqueName: \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\") pod \"4f7c2061-2499-496f-ad9a-118aeff161ea\" (UID: \"4f7c2061-2499-496f-ad9a-118aeff161ea\") "
Nov 03 22:30:20.470442 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.470412    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume" pod "4f7c2061-2499-496f-ad9a-118aeff161ea" (UID: "4f7c2061-2499-496f-ad9a-118aeff161ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:20.472227 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.472203    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l" pod "4f7c2061-2499-496f-ad9a-118aeff161ea" (UID: "4f7c2061-2499-496f-ad9a-118aeff161ea"). InnerVolumeSpecName "kube-api-access-k9r6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:20.510890 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.510870    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:30:20.570732 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.570694    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\" (UniqueName: \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\") pod \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") "
Nov 03 22:30:20.570844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.570775    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\") pod \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\" (UID: \"4b26ebae-b2da-49d5-b49b-7f3119c64c8d\") "
Nov 03 22:30:20.571017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.570998    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9r6l\" (UniqueName: \"kubernetes.io/projected/4f7c2061-2499-496f-ad9a-118aeff161ea-kube-api-access-k9r6l\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:20.571115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.571021    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7c2061-2499-496f-ad9a-118aeff161ea-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:20.571233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.571206    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume" pod "4b26ebae-b2da-49d5-b49b-7f3119c64c8d" (UID: "4b26ebae-b2da-49d5-b49b-7f3119c64c8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:20.572664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.572627    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9" pod "4b26ebae-b2da-49d5-b49b-7f3119c64c8d" (UID: "4b26ebae-b2da-49d5-b49b-7f3119c64c8d"). InnerVolumeSpecName "kube-api-access-gnfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:20.671705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.671681    5533 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-plugins-config-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:20.671705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.671704    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfs9\" (UniqueName: \"kubernetes.io/projected/4b26ebae-b2da-49d5-b49b-7f3119c64c8d-kube-api-access-gnfs9\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:20.762801 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.762778    5533 generic.go:356] "Generic (PLEG): container finished" podID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerID="5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1" exitCode=0
Nov 03 22:30:20.762919 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.762833    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"
Nov 03 22:30:20.762919 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.762867    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" event={"ID":"4b26ebae-b2da-49d5-b49b-7f3119c64c8d","Type":"ContainerDied","Data":"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"}
Nov 03 22:30:20.763056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.762921    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d" event={"ID":"4b26ebae-b2da-49d5-b49b-7f3119c64c8d","Type":"ContainerDied","Data":"1a268ac351e2ab4ada13310e7662690c51568086b23404ef53f57d4a524755b9"}
Nov 03 22:30:20.763056 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.762949    5533 scope.go:122] "RemoveContainer" containerID="5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"
Nov 03 22:30:20.764678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.764387    5533 generic.go:356] "Generic (PLEG): container finished" podID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerID="e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a" exitCode=0
Nov 03 22:30:20.764678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.764451    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"
Nov 03 22:30:20.764678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.764472    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" event={"ID":"4f7c2061-2499-496f-ad9a-118aeff161ea","Type":"ContainerDied","Data":"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"}
Nov 03 22:30:20.764678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.764513    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4" event={"ID":"4f7c2061-2499-496f-ad9a-118aeff161ea","Type":"ContainerDied","Data":"e6f9b71ab5179f6a22bdc8ae48f182dcafd52240664be145d663ec0a72fe86ad"}
Nov 03 22:30:20.772288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.772268    5533 scope.go:122] "RemoveContainer" containerID="5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"
Nov 03 22:30:20.772547 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:30:20.772528    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1\": container with ID starting with 5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1 not found: ID does not exist" containerID="5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"
Nov 03 22:30:20.772626 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.772557    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1"} err="failed to get container status \"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1\": rpc error: code = NotFound desc = could not find container \"5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1\": container with ID starting with 5d33ec65b5e60607a585490ec06a671fb016f2af78e996cff97cdcea78b327c1 not found: ID does not exist"
Nov 03 22:30:20.772626 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.772580    5533 scope.go:122] "RemoveContainer" containerID="e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"
Nov 03 22:30:20.780485 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.780470    5533 scope.go:122] "RemoveContainer" containerID="e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"
Nov 03 22:30:20.780763 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:30:20.780747    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a\": container with ID starting with e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a not found: ID does not exist" containerID="e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"
Nov 03 22:30:20.780831 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.780772    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a"} err="failed to get container status \"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a\": rpc error: code = NotFound desc = could not find container \"e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a\": container with ID starting with e9375aef2964a5ea45e9f4ccfa3f04b973275bdb423deb36f85585361a39e90a not found: ID does not exist"
Nov 03 22:30:20.917344 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.917318    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:30:20.923495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:20.923443    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-51b683-76f6dc7d9vjdx4"]
Nov 03 22:30:21.027072 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:21.027042    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:30:21.031220 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:21.031197    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-51b683-9cd9d6rs9d"]
Nov 03 22:30:22.491556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:22.491521    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" path="/var/lib/kubelet/pods/4b26ebae-b2da-49d5-b49b-7f3119c64c8d/volumes"
Nov 03 22:30:22.492027 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:22.492010    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" path="/var/lib/kubelet/pods/4f7c2061-2499-496f-ad9a-118aeff161ea/volumes"
Nov 03 22:30:22.877080 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:22.877012    5533 scope.go:122] "RemoveContainer" containerID="830d26deca06e4c127a44db792778d7f039cad99b003c9ae93df35ca722f8352"
Nov 03 22:30:22.885300 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:22.885282    5533 scope.go:122] "RemoveContainer" containerID="fa2b4b1e737dda8998fe5ce6db10c4abced9784cce2db3ca0c312ec764352a7f"
Nov 03 22:30:36.242433 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.242395    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"]
Nov 03 22:30:36.288816 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.288785    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5x82x"]
Nov 03 22:30:36.369340 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369304    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"]
Nov 03 22:30:36.369718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369702    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing"
Nov 03 22:30:36.369718 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369717    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369742    5533 cpu_manager.go:414] "containerMap: removing container" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369748    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369756    5533 cpu_manager.go:414] "containerMap: removing container" podUID="7c49001a-4a42-4b7b-9770-58510f222e4b" containerName="cleanup"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369761    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c49001a-4a42-4b7b-9770-58510f222e4b" containerName="cleanup"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369825    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b26ebae-b2da-49d5-b49b-7f3119c64c8d" containerName="payload-pre-processing"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369837    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7c2061-2499-496f-ad9a-118aeff161ea" containerName="payload-processing"
Nov 03 22:30:36.369906 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.369843    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c49001a-4a42-4b7b-9770-58510f222e4b" containerName="cleanup"
Nov 03 22:30:36.374881 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.374859    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.397419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.397392    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.397529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.397475    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8st\" (UniqueName: \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.397605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.397589    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.427328 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.427305    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"]
Nov 03 22:30:36.467737 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.467717    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:30:36.467857 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.467746    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:30:36.492241 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.492218    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df09764-2383-4b14-a7e3-7353795714be" path="/var/lib/kubelet/pods/0df09764-2383-4b14-a7e3-7353795714be/volumes"
Nov 03 22:30:36.498445 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.498399    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.498541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.498455    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.498541 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.498495    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8st\" (UniqueName: \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.499073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.499048    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.500779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.500760    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.556004 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.555980    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8st\" (UniqueName: \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\") pod \"maas-api-revoke-sub-3d4156f1-mkp49\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:36.688601 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:36.688576    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:37.053491 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:37.053415    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"]
Nov 03 22:30:37.056114 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:30:37.056070    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac6a63e_9b59_4a43_ae05_296bcd3068f6.slice/crio-b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344 WatchSource:0}: Error finding container b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344: Status 404 returned error can't find the container with id b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344
Nov 03 22:30:37.862128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:37.862096    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49" event={"ID":"8ac6a63e-9b59-4a43-ae05-296bcd3068f6","Type":"ContainerStarted","Data":"f38791331d2327c76c02cfe38ff73b3df23b3bd2a5bd55c1dca8b7b759d5cfae"}
Nov 03 22:30:37.862493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:37.862152    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49" event={"ID":"8ac6a63e-9b59-4a43-ae05-296bcd3068f6","Type":"ContainerStarted","Data":"b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344"}
Nov 03 22:30:37.972410 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:37.972356    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49" podStartSLOduration=1.9723394010000002 podStartE2EDuration="1.972339401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:37.971698031 +0000 UTC m=+6139.603011505" watchObservedRunningTime="2026-11-03 22:30:37.972339401 +0000 UTC m=+6139.603652879"
Nov 03 22:30:38.869950 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:38.869916    5533 generic.go:356] "Generic (PLEG): container finished" podID="8ac6a63e-9b59-4a43-ae05-296bcd3068f6" containerID="f38791331d2327c76c02cfe38ff73b3df23b3bd2a5bd55c1dca8b7b759d5cfae" exitCode=0
Nov 03 22:30:38.870464 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:38.870007    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49" event={"ID":"8ac6a63e-9b59-4a43-ae05-296bcd3068f6","Type":"ContainerDied","Data":"f38791331d2327c76c02cfe38ff73b3df23b3bd2a5bd55c1dca8b7b759d5cfae"}
Nov 03 22:30:40.006143 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.006117    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:40.133225 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.133171    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\" (UniqueName: \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\") pod \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") "
Nov 03 22:30:40.133225 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.133211    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\") pod \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") "
Nov 03 22:30:40.133413 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.133268    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\") pod \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\" (UID: \"8ac6a63e-9b59-4a43-ae05-296bcd3068f6\") "
Nov 03 22:30:40.133674 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.133628    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca" pod "8ac6a63e-9b59-4a43-ae05-296bcd3068f6" (UID: "8ac6a63e-9b59-4a43-ae05-296bcd3068f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:30:40.135363 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.135337    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st" pod "8ac6a63e-9b59-4a43-ae05-296bcd3068f6" (UID: "8ac6a63e-9b59-4a43-ae05-296bcd3068f6"). InnerVolumeSpecName "kube-api-access-lj8st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:40.135775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.135758    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca" pod "8ac6a63e-9b59-4a43-ae05-296bcd3068f6" (UID: "8ac6a63e-9b59-4a43-ae05-296bcd3068f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:30:40.234429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.234400    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8st\" (UniqueName: \"kubernetes.io/projected/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-kube-api-access-lj8st\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:40.234429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.234425    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:40.234578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.234435    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac6a63e-9b59-4a43-ae05-296bcd3068f6-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:40.882190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.882120    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"
Nov 03 22:30:40.882190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.882145    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49" event={"ID":"8ac6a63e-9b59-4a43-ae05-296bcd3068f6","Type":"ContainerDied","Data":"b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344"}
Nov 03 22:30:40.882362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:40.882200    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d3810ce14772e3c081e7437940cf15baa077d812695caeda04d722a8a26344"
Nov 03 22:30:41.045070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:41.045041    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"]
Nov 03 22:30:41.047493 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:41.047471    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dpmzk"]
Nov 03 22:30:42.492162 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:42.492123    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad3a441-fbd1-4c87-b071-8abd4e91fcb4" path="/var/lib/kubelet/pods/7ad3a441-fbd1-4c87-b071-8abd4e91fcb4/volumes"
Nov 03 22:30:49.261073 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:30:49.261027    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:30:51.102712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:51.102678    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:30:53.109660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:53.108862    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:30:53.109660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:53.108952    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:53.145757 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:53.145730    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:30:53.145898 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:53.145785    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:30:55.020827 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:55.020795    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 22:30:55.021239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:55.020987    5533 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" podUID="e93933ed-7935-46bf-8a67-258fda1d0fd6" containerName="manager" containerID="cri-o://2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba" gracePeriod=10
Nov 03 22:30:56.591332 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.591299    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 22:30:56.679575 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.679550    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\" (UniqueName: \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\") pod \"e93933ed-7935-46bf-8a67-258fda1d0fd6\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") "
Nov 03 22:30:56.679734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.679604    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\") pod \"e93933ed-7935-46bf-8a67-258fda1d0fd6\" (UID: \"e93933ed-7935-46bf-8a67-258fda1d0fd6\") "
Nov 03 22:30:56.679921 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.679901    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume" pod "e93933ed-7935-46bf-8a67-258fda1d0fd6" (UID: "e93933ed-7935-46bf-8a67-258fda1d0fd6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:30:56.681427 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.681406    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t" pod "e93933ed-7935-46bf-8a67-258fda1d0fd6" (UID: "e93933ed-7935-46bf-8a67-258fda1d0fd6"). InnerVolumeSpecName "kube-api-access-wsn6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:30:56.781003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.780978    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsn6t\" (UniqueName: \"kubernetes.io/projected/e93933ed-7935-46bf-8a67-258fda1d0fd6-kube-api-access-wsn6t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:56.781003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.781000    5533 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93933ed-7935-46bf-8a67-258fda1d0fd6-extensions-socket-volume\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:30:56.976910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.976836    5533 generic.go:356] "Generic (PLEG): container finished" podID="e93933ed-7935-46bf-8a67-258fda1d0fd6" containerID="2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba" exitCode=0
Nov 03 22:30:56.976910 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.976892    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"
Nov 03 22:30:56.977088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.976915    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" event={"ID":"e93933ed-7935-46bf-8a67-258fda1d0fd6","Type":"ContainerDied","Data":"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"}
Nov 03 22:30:56.977088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.976967    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t" event={"ID":"e93933ed-7935-46bf-8a67-258fda1d0fd6","Type":"ContainerDied","Data":"3cde68ced461b9a65333f2eafcf813b52d755f7985c2d0abb9a4701749cfc603"}
Nov 03 22:30:56.977088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.976989    5533 scope.go:122] "RemoveContainer" containerID="2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"
Nov 03 22:30:56.990878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.990859    5533 scope.go:122] "RemoveContainer" containerID="2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"
Nov 03 22:30:56.991152 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:30:56.991133    5533 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba\": container with ID starting with 2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba not found: ID does not exist" containerID="2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"
Nov 03 22:30:56.991211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:56.991160    5533 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba"} err="failed to get container status \"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba\": rpc error: code = NotFound desc = could not find container \"2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba\": container with ID starting with 2344b3fa84daed79d7518eb098f7fa7b9a841c6ed17f4d9e32311d4ef17315ba not found: ID does not exist"
Nov 03 22:30:57.132858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:57.132833    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 22:30:57.137132 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:57.137110    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gn82t"]
Nov 03 22:30:58.492712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:30:58.492678    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93933ed-7935-46bf-8a67-258fda1d0fd6" path="/var/lib/kubelet/pods/e93933ed-7935-46bf-8a67-258fda1d0fd6/volumes"
Nov 03 22:31:02.046501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:02.046458    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"]
Nov 03 22:31:02.093193 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:02.093162    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-nb498"]
Nov 03 22:31:02.491490 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:02.491459    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21629a0-f620-47d3-8755-b43604397ee0" path="/var/lib/kubelet/pods/c21629a0-f620-47d3-8755-b43604397ee0/volumes"
Nov 03 22:31:19.046407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:19.046369    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"]
Nov 03 22:31:19.092828 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:19.092799    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tlv9x"]
Nov 03 22:31:19.260164 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:31:19.260131    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:31:20.491414 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:20.491379    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e748ffe-fba6-40a2-bdda-1c53c1eb906d" path="/var/lib/kubelet/pods/7e748ffe-fba6-40a2-bdda-1c53c1eb906d/volumes"
Nov 03 22:31:22.921763 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:22.921730    5533 scope.go:122] "RemoveContainer" containerID="dc691aaaa360e7f0fbc819b56528f03fe111bb3a165a403029edcfea35266aa6"
Nov 03 22:31:22.930782 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:22.930760    5533 scope.go:122] "RemoveContainer" containerID="926d6a40f451717575a0aa0c3b567d94c82d388426463a712ad9d30e8bd70ecf"
Nov 03 22:31:22.939215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:22.939187    5533 scope.go:122] "RemoveContainer" containerID="615ae8e10088a3fb7f45b44e9a42afddd99413951ac6958bdcc74f10caae74bc"
Nov 03 22:31:49.260415 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:31:49.260380    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:31:58.553086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553048    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"]
Nov 03 22:31:58.553470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553403    5533 cpu_manager.go:414] "containerMap: removing container" podUID="8ac6a63e-9b59-4a43-ae05-296bcd3068f6" containerName="revoke-keys"
Nov 03 22:31:58.553470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553413    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac6a63e-9b59-4a43-ae05-296bcd3068f6" containerName="revoke-keys"
Nov 03 22:31:58.553470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553432    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e93933ed-7935-46bf-8a67-258fda1d0fd6" containerName="manager"
Nov 03 22:31:58.553470 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553437    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93933ed-7935-46bf-8a67-258fda1d0fd6" containerName="manager"
Nov 03 22:31:58.553621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553494    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac6a63e-9b59-4a43-ae05-296bcd3068f6" containerName="revoke-keys"
Nov 03 22:31:58.553621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.553505    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93933ed-7935-46bf-8a67-258fda1d0fd6" containerName="manager"
Nov 03 22:31:58.556808 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.556789    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.566261 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.566237    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c0ddb3-f9d6-42db-83d6-304fc86bc632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.566480 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.566458    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9qz\" (UniqueName: \"kubernetes.io/projected/94c0ddb3-f9d6-42db-83d6-304fc86bc632-kube-api-access-zr9qz\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.658630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.653346    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vqxqx\""
Nov 03 22:31:58.658630 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.657318    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"]
Nov 03 22:31:58.667404 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.667379    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c0ddb3-f9d6-42db-83d6-304fc86bc632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.667514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.667424    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9qz\" (UniqueName: \"kubernetes.io/projected/94c0ddb3-f9d6-42db-83d6-304fc86bc632-kube-api-access-zr9qz\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.667758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.667739    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94c0ddb3-f9d6-42db-83d6-304fc86bc632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.725684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.725661    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9qz\" (UniqueName: \"kubernetes.io/projected/94c0ddb3-f9d6-42db-83d6-304fc86bc632-kube-api-access-zr9qz\") pod \"kuadrant-operator-controller-manager-678d87766f-2drq5\" (UID: \"94c0ddb3-f9d6-42db-83d6-304fc86bc632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:58.869561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:58.869493    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:31:59.304658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:31:59.300922    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"]
Nov 03 22:31:59.305259 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:31:59.305217    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c0ddb3_f9d6_42db_83d6_304fc86bc632.slice/crio-d2a7bddcb128cead0e7d77cf037c84c915ebfa4982e7751f1de0fe5d4475d423 WatchSource:0}: Error finding container d2a7bddcb128cead0e7d77cf037c84c915ebfa4982e7751f1de0fe5d4475d423: Status 404 returned error can't find the container with id d2a7bddcb128cead0e7d77cf037c84c915ebfa4982e7751f1de0fe5d4475d423
Nov 03 22:32:00.318042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:00.318005    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5" event={"ID":"94c0ddb3-f9d6-42db-83d6-304fc86bc632","Type":"ContainerStarted","Data":"877ac83617a329441fe19fb7eca1e728cbd23540d7f163d812ce551abf892ecd"}
Nov 03 22:32:00.318444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:00.318062    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5" event={"ID":"94c0ddb3-f9d6-42db-83d6-304fc86bc632","Type":"ContainerStarted","Data":"d2a7bddcb128cead0e7d77cf037c84c915ebfa4982e7751f1de0fe5d4475d423"}
Nov 03 22:32:00.318444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:00.318091    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:32:00.529303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:00.529248    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5" podStartSLOduration=2.529236504 podStartE2EDuration="2.529236504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:00.482156162 +0000 UTC m=+6222.113469640" watchObservedRunningTime="2026-11-03 22:32:00.529236504 +0000 UTC m=+6222.160549980"
Nov 03 22:32:09.374363 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:09.374327    5533 generic.go:356] "Generic (PLEG): container finished" podID="7e9c8826-0e09-4d08-836c-186eab92e73e" containerID="633055ff35614bde497982c504536cef3ab5ca458f352960ed63011aa1e81d6b" exitCode=1
Nov 03 22:32:09.374811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:09.374402    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-rzlw2" event={"ID":"7e9c8826-0e09-4d08-836c-186eab92e73e","Type":"ContainerDied","Data":"633055ff35614bde497982c504536cef3ab5ca458f352960ed63011aa1e81d6b"}
Nov 03 22:32:09.374811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:09.374763    5533 scope.go:122] "RemoveContainer" containerID="633055ff35614bde497982c504536cef3ab5ca458f352960ed63011aa1e81d6b"
Nov 03 22:32:09.375124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:09.375107    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:32:09.794530 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:09.794501    5533 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:32:10.438556 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:10.438529    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 22:32:11.324728 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:11.324700    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2drq5"
Nov 03 22:32:11.386313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:11.386286    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77b845648-rzlw2" event={"ID":"7e9c8826-0e09-4d08-836c-186eab92e73e","Type":"ContainerStarted","Data":"b184f917a5cf06312c028a36077ef2586acc4b625ef451fad5283cc6e7f85894"}
Nov 03 22:32:11.386455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:11.386420    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:32:19.265081 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:32:19.265039    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:32:22.393407 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:22.393374    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77b845648-rzlw2"
Nov 03 22:32:22.974291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:22.974256    5533 scope.go:122] "RemoveContainer" containerID="46de56bf40246c818b16e824aeecb354e5042d24ac37e31bcbf2dff24bc06d47"
Nov 03 22:32:22.982877 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:22.982854    5533 scope.go:122] "RemoveContainer" containerID="ef9230414849482bacbfa539530f9c1489fee37e6bdd2ff03e9d609b1e4ddcff"
Nov 03 22:32:22.990485 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:22.990465    5533 scope.go:122] "RemoveContainer" containerID="7d17e7880e67f62ae8d39f6d9d366330b06df736e5fc9de3d399743730e49927"
Nov 03 22:32:23.066019 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:23.065993    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:32:33.112472 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.112434    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"]
Nov 03 22:32:33.117736 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.117716    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.168604 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.168570    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:32:33.215430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.215404    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:32:33.217455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.217434    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"]
Nov 03 22:32:33.242089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.242065    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzsp\" (UniqueName: \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.242213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.242121    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.242269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.242243    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.343171 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.343133    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.343350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.343200    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.343350 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.343252    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzsp\" (UniqueName: \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.343897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.343870    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.345466 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.345443    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.401184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.401122    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzsp\" (UniqueName: \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\") pod \"maas-api-revoke-sub-d576e68b-hh9f4\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.430436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.430415    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:33.796815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:33.796786    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"]
Nov 03 22:32:33.798692 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:32:33.798659    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cc89da_041b_47cf_addb_4febb0fd9b0e.slice/crio-b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470 WatchSource:0}: Error finding container b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470: Status 404 returned error can't find the container with id b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470
Nov 03 22:32:34.513833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:34.513801    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4" event={"ID":"61cc89da-041b-47cf-addb-4febb0fd9b0e","Type":"ContainerStarted","Data":"92c0f5994483d4f5e5cd303fbf8dbaa6d3effb6bd4312c63199ea44c09fbadff"}
Nov 03 22:32:34.514202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:34.513856    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4" event={"ID":"61cc89da-041b-47cf-addb-4febb0fd9b0e","Type":"ContainerStarted","Data":"b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470"}
Nov 03 22:32:34.624126 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:34.624075    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4" podStartSLOduration=1.624061969 podStartE2EDuration="1.624061969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:34.622068616 +0000 UTC m=+6256.253382093" watchObservedRunningTime="2026-11-03 22:32:34.624061969 +0000 UTC m=+6256.255375471"
Nov 03 22:32:35.521288 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:35.521257    5533 generic.go:356] "Generic (PLEG): container finished" podID="61cc89da-041b-47cf-addb-4febb0fd9b0e" containerID="92c0f5994483d4f5e5cd303fbf8dbaa6d3effb6bd4312c63199ea44c09fbadff" exitCode=0
Nov 03 22:32:35.521753 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:35.521327    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4" event={"ID":"61cc89da-041b-47cf-addb-4febb0fd9b0e","Type":"ContainerDied","Data":"92c0f5994483d4f5e5cd303fbf8dbaa6d3effb6bd4312c63199ea44c09fbadff"}
Nov 03 22:32:36.658982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.658952    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:36.778620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.778560    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\") pod \"61cc89da-041b-47cf-addb-4febb0fd9b0e\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") "
Nov 03 22:32:36.778620 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.778603    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\" (UniqueName: \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\") pod \"61cc89da-041b-47cf-addb-4febb0fd9b0e\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") "
Nov 03 22:32:36.778859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.778646    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\") pod \"61cc89da-041b-47cf-addb-4febb0fd9b0e\" (UID: \"61cc89da-041b-47cf-addb-4febb0fd9b0e\") "
Nov 03 22:32:36.779068 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.779036    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca" pod "61cc89da-041b-47cf-addb-4febb0fd9b0e" (UID: "61cc89da-041b-47cf-addb-4febb0fd9b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:32:36.780753 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.780722    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca" pod "61cc89da-041b-47cf-addb-4febb0fd9b0e" (UID: "61cc89da-041b-47cf-addb-4febb0fd9b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:32:36.780851 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.780826    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp" pod "61cc89da-041b-47cf-addb-4febb0fd9b0e" (UID: "61cc89da-041b-47cf-addb-4febb0fd9b0e"). InnerVolumeSpecName "kube-api-access-4fzsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:32:36.879458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.879434    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:36.879458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.879458    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzsp\" (UniqueName: \"kubernetes.io/projected/61cc89da-041b-47cf-addb-4febb0fd9b0e-kube-api-access-4fzsp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:36.879583 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:36.879468    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61cc89da-041b-47cf-addb-4febb0fd9b0e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:37.045202 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:37.045111    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"]
Nov 03 22:32:37.048141 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:37.048114    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g2cgk"]
Nov 03 22:32:37.534454 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:37.534416    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4" event={"ID":"61cc89da-041b-47cf-addb-4febb0fd9b0e","Type":"ContainerDied","Data":"b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470"}
Nov 03 22:32:37.534683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:37.534478    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93912cda0b05be7be21b7237688092fb07c6024acdaacc86885f5edccfcd470"
Nov 03 22:32:37.534683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:37.534423    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"
Nov 03 22:32:38.491482 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:38.491446    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0751a82-bc21-4014-8fa9-0790099f7493" path="/var/lib/kubelet/pods/f0751a82-bc21-4014-8fa9-0790099f7493/volumes"
Nov 03 22:32:45.888471 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:45.888432    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:32:48.045031 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:48.044993    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"]
Nov 03 22:32:48.093659 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:48.093477    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2czg"]
Nov 03 22:32:48.492233 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:48.492204    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695a39dc-648b-43f9-a4ce-2c4e22c149f6" path="/var/lib/kubelet/pods/695a39dc-648b-43f9-a4ce-2c4e22c149f6/volumes"
Nov 03 22:32:49.260410 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:32:49.260373    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:32:54.498112 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.498077    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"]
Nov 03 22:32:54.498820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.498456    5533 cpu_manager.go:414] "containerMap: removing container" podUID="61cc89da-041b-47cf-addb-4febb0fd9b0e" containerName="revoke-keys"
Nov 03 22:32:54.498820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.498467    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cc89da-041b-47cf-addb-4febb0fd9b0e" containerName="revoke-keys"
Nov 03 22:32:54.498820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.498529    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cc89da-041b-47cf-addb-4febb0fd9b0e" containerName="revoke-keys"
Nov 03 22:32:54.501534 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.501516    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.560169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.560147    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:32:54.606821 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.606793    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:32:54.608907 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.608885    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"]
Nov 03 22:32:54.627239 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.627213    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.627347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.627251    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvbw\" (UniqueName: \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.627347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.627282    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.728901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.728861    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.728901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.728897    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvbw\" (UniqueName: \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.729148 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.728916    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.729728 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.729701    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.731312 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.731288    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.787334 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.787271    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvbw\" (UniqueName: \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\") pod \"maas-api-revoke-sub-d2bc93bc-2bwms\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:54.813914 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:54.813885    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:55.183218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:55.183193    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"]
Nov 03 22:32:55.185458 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:32:55.185428    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fc0495_071c_499e_907c_f064006b5cc5.slice/crio-1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef WatchSource:0}: Error finding container 1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef: Status 404 returned error can't find the container with id 1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef
Nov 03 22:32:55.302409 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:55.302378    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:32:55.648181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:55.648106    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms" event={"ID":"c8fc0495-071c-499e-907c-f064006b5cc5","Type":"ContainerStarted","Data":"255c2eed80a6c7471017c65b81bd407c67aaf5f5427a9bd84605813f65c670fc"}
Nov 03 22:32:55.648181 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:55.648159    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms" event={"ID":"c8fc0495-071c-499e-907c-f064006b5cc5","Type":"ContainerStarted","Data":"1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef"}
Nov 03 22:32:55.757383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:55.757334    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms" podStartSLOduration=1.757317605 podStartE2EDuration="1.757317605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:55.756732947 +0000 UTC m=+6277.388046423" watchObservedRunningTime="2026-11-03 22:32:55.757317605 +0000 UTC m=+6277.388631080"
Nov 03 22:32:56.654813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:56.654780    5533 generic.go:356] "Generic (PLEG): container finished" podID="c8fc0495-071c-499e-907c-f064006b5cc5" containerID="255c2eed80a6c7471017c65b81bd407c67aaf5f5427a9bd84605813f65c670fc" exitCode=0
Nov 03 22:32:56.655311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:56.654839    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms" event={"ID":"c8fc0495-071c-499e-907c-f064006b5cc5","Type":"ContainerDied","Data":"255c2eed80a6c7471017c65b81bd407c67aaf5f5427a9bd84605813f65c670fc"}
Nov 03 22:32:57.785535 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.785511    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:57.850930 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.850902    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\") pod \"c8fc0495-071c-499e-907c-f064006b5cc5\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") "
Nov 03 22:32:57.851054 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.850972    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\" (UniqueName: \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\") pod \"c8fc0495-071c-499e-907c-f064006b5cc5\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") "
Nov 03 22:32:57.852881 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.852853    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw" pod "c8fc0495-071c-499e-907c-f064006b5cc5" (UID: "c8fc0495-071c-499e-907c-f064006b5cc5"). InnerVolumeSpecName "kube-api-access-kpvbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:32:57.852996 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.852882    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca" pod "c8fc0495-071c-499e-907c-f064006b5cc5" (UID: "c8fc0495-071c-499e-907c-f064006b5cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:32:57.951664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.951626    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\") pod \"c8fc0495-071c-499e-907c-f064006b5cc5\" (UID: \"c8fc0495-071c-499e-907c-f064006b5cc5\") "
Nov 03 22:32:57.951868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.951853    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvbw\" (UniqueName: \"kubernetes.io/projected/c8fc0495-071c-499e-907c-f064006b5cc5-kube-api-access-kpvbw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:57.951923 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.951873    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:57.951963 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:57.951916    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca" pod "c8fc0495-071c-499e-907c-f064006b5cc5" (UID: "c8fc0495-071c-499e-907c-f064006b5cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:32:58.052719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:58.052693    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fc0495-071c-499e-907c-f064006b5cc5-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:32:58.666506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:58.666480    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"
Nov 03 22:32:58.666683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:58.666514    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms" event={"ID":"c8fc0495-071c-499e-907c-f064006b5cc5","Type":"ContainerDied","Data":"1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef"}
Nov 03 22:32:58.666683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:32:58.666564    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d821b4a55825ab0cafff81030486a480fafadd27ca00750aadc59e11e85b8ef"
Nov 03 22:33:03.410946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:03.410907    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:33:19.260745 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:33:19.260712    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:33:19.607794 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:19.607708    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:33:19.619791 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:19.619768    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:33:19.659514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:19.659490    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:33:19.672361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:19.672341    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:33:19.973656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:19.973603    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:33:23.026606 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.026569    5533 scope.go:122] "RemoveContainer" containerID="23f11a3956c67d717be9217c805285b2aa8c94e364da994be29019e79e18866c"
Nov 03 22:33:23.038478 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.038459    5533 scope.go:122] "RemoveContainer" containerID="8205d0b67b863e158351139a00d9a40785b162e41381e4c4f46c107df4a19b52"
Nov 03 22:33:23.121838 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.121809    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:33:23.122087 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.122057    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:33:23.148781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.148757    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:33:23.148911 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:23.148811    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:33:32.289492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.289460    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"]
Nov 03 22:33:32.290014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.289836    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c8fc0495-071c-499e-907c-f064006b5cc5" containerName="revoke-keys"
Nov 03 22:33:32.290014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.289847    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fc0495-071c-499e-907c-f064006b5cc5" containerName="revoke-keys"
Nov 03 22:33:32.290014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.289916    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fc0495-071c-499e-907c-f064006b5cc5" containerName="revoke-keys"
Nov 03 22:33:32.294458 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.294438    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.371357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.371318    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.371504 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.371372    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp67m\" (UniqueName: \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.371557 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.371504    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.391735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.391713    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:33:32.391735 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.391726    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:33:32.393736 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.393712    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"]
Nov 03 22:33:32.472798 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.472773    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp67m\" (UniqueName: \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.472900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.472806    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.472900 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.472879    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.473451 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.473430    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.475194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.475173    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.530768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.530738    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp67m\" (UniqueName: \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\") pod \"maas-api-revoke-sub-b03a9a92-5kwld\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.607168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.607119    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:32.969920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:32.969896    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"]
Nov 03 22:33:32.971717 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:33:32.971688    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5816f8_a46c_4050_9a11_97de66b7f911.slice/crio-94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963 WatchSource:0}: Error finding container 94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963: Status 404 returned error can't find the container with id 94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963
Nov 03 22:33:33.883960 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:33.883930    5533 generic.go:356] "Generic (PLEG): container finished" podID="5e5816f8-a46c-4050-9a11-97de66b7f911" containerID="88c3f93153d58c43983a5f2e9cc8c6dca1dbc40fd63e021604838c0bd860c252" exitCode=0
Nov 03 22:33:33.884281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:33.884010    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld" event={"ID":"5e5816f8-a46c-4050-9a11-97de66b7f911","Type":"ContainerDied","Data":"88c3f93153d58c43983a5f2e9cc8c6dca1dbc40fd63e021604838c0bd860c252"}
Nov 03 22:33:33.884281 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:33.884060    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld" event={"ID":"5e5816f8-a46c-4050-9a11-97de66b7f911","Type":"ContainerStarted","Data":"94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963"}
Nov 03 22:33:35.013400 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.013374    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:35.095224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.095196    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\") pod \"5e5816f8-a46c-4050-9a11-97de66b7f911\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") "
Nov 03 22:33:35.095365 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.095235    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\" (UniqueName: \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\") pod \"5e5816f8-a46c-4050-9a11-97de66b7f911\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") "
Nov 03 22:33:35.095365 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.095314    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\") pod \"5e5816f8-a46c-4050-9a11-97de66b7f911\" (UID: \"5e5816f8-a46c-4050-9a11-97de66b7f911\") "
Nov 03 22:33:35.095563 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.095540    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca" pod "5e5816f8-a46c-4050-9a11-97de66b7f911" (UID: "5e5816f8-a46c-4050-9a11-97de66b7f911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:33:35.095688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.095670    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:35.097293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.097272    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m" pod "5e5816f8-a46c-4050-9a11-97de66b7f911" (UID: "5e5816f8-a46c-4050-9a11-97de66b7f911"). InnerVolumeSpecName "kube-api-access-rp67m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:33:35.097361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.097343    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca" pod "5e5816f8-a46c-4050-9a11-97de66b7f911" (UID: "5e5816f8-a46c-4050-9a11-97de66b7f911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:33:35.196510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.196484    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp67m\" (UniqueName: \"kubernetes.io/projected/5e5816f8-a46c-4050-9a11-97de66b7f911-kube-api-access-rp67m\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:35.196510 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.196509    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5816f8-a46c-4050-9a11-97de66b7f911-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:35.895180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.895152    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"
Nov 03 22:33:35.895383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.895188    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld" event={"ID":"5e5816f8-a46c-4050-9a11-97de66b7f911","Type":"ContainerDied","Data":"94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963"}
Nov 03 22:33:35.895383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:35.895244    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94dedf6ec53e16e49400b3c07404846fa60d4e65f615e03156032277a8adc963"
Nov 03 22:33:43.103253 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.103217    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"]
Nov 03 22:33:43.103849 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.103824    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5e5816f8-a46c-4050-9a11-97de66b7f911" containerName="revoke-keys"
Nov 03 22:33:43.103915 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.103853    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5816f8-a46c-4050-9a11-97de66b7f911" containerName="revoke-keys"
Nov 03 22:33:43.104049 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.104033    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5816f8-a46c-4050-9a11-97de66b7f911" containerName="revoke-keys"
Nov 03 22:33:43.107540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.107522    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.157429 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.157390    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jz7\" (UniqueName: \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.157699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.157470    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.157699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.157619    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.157699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.157678    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:33:43.204312 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.204261    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:33:43.206647 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.206610    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"]
Nov 03 22:33:43.258447 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.258421    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.258538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.258472    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jz7\" (UniqueName: \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.258589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.258577    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.259184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.259160    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.260897 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.260872    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.316810 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.316781    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jz7\" (UniqueName: \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\") pod \"maas-api-revoke-sub-3fb908d3-9zq2z\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.420712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.420689    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:43.785063 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.785037    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"]
Nov 03 22:33:43.787135 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:33:43.787106    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4db16_5bb2_4a8b_9225_921f7dba147b.slice/crio-e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a WatchSource:0}: Error finding container e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a: Status 404 returned error can't find the container with id e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a
Nov 03 22:33:43.940459 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:43.940429    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z" event={"ID":"3bf4db16-5bb2-4a8b-9225-921f7dba147b","Type":"ContainerStarted","Data":"e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a"}
Nov 03 22:33:44.947264 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:44.947231    5533 generic.go:356] "Generic (PLEG): container finished" podID="3bf4db16-5bb2-4a8b-9225-921f7dba147b" containerID="453c12142551dfcace36e577a3ebece8e6de4a5ec0cc1d838e158e38c25847da" exitCode=0
Nov 03 22:33:44.947669 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:44.947275    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z" event={"ID":"3bf4db16-5bb2-4a8b-9225-921f7dba147b","Type":"ContainerDied","Data":"453c12142551dfcace36e577a3ebece8e6de4a5ec0cc1d838e158e38c25847da"}
Nov 03 22:33:46.099661 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.099547    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:46.181820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.181778    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\") pod \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") "
Nov 03 22:33:46.182024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.181859    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\") pod \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") "
Nov 03 22:33:46.182024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.181987    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\" (UniqueName: \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\") pod \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\" (UID: \"3bf4db16-5bb2-4a8b-9225-921f7dba147b\") "
Nov 03 22:33:46.182156 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.182125    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca" pod "3bf4db16-5bb2-4a8b-9225-921f7dba147b" (UID: "3bf4db16-5bb2-4a8b-9225-921f7dba147b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:33:46.182305 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.182287    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:46.183935 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.183903    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca" pod "3bf4db16-5bb2-4a8b-9225-921f7dba147b" (UID: "3bf4db16-5bb2-4a8b-9225-921f7dba147b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:33:46.184064 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.183960    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7" pod "3bf4db16-5bb2-4a8b-9225-921f7dba147b" (UID: "3bf4db16-5bb2-4a8b-9225-921f7dba147b"). InnerVolumeSpecName "kube-api-access-58jz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:33:46.283570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.283498    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jz7\" (UniqueName: \"kubernetes.io/projected/3bf4db16-5bb2-4a8b-9225-921f7dba147b-kube-api-access-58jz7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:46.283570 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.283539    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf4db16-5bb2-4a8b-9225-921f7dba147b-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:33:46.958903 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.958869    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"
Nov 03 22:33:46.959097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.958919    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z" event={"ID":"3bf4db16-5bb2-4a8b-9225-921f7dba147b","Type":"ContainerDied","Data":"e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a"}
Nov 03 22:33:46.959097 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:46.959012    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e70849cd1abbb2dfb9d26b1238b4e5ff804b00bea69f0631eace4892500fea2a"
Nov 03 22:33:49.260044 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:33:49.260010    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:33:55.862502 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:33:55.862466    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:34:07.762081 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:07.762045    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:34:16.467360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.467321    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"]
Nov 03 22:34:16.467774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.467713    5533 cpu_manager.go:414] "containerMap: removing container" podUID="3bf4db16-5bb2-4a8b-9225-921f7dba147b" containerName="revoke-keys"
Nov 03 22:34:16.467774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.467726    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf4db16-5bb2-4a8b-9225-921f7dba147b" containerName="revoke-keys"
Nov 03 22:34:16.467878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.467808    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf4db16-5bb2-4a8b-9225-921f7dba147b" containerName="revoke-keys"
Nov 03 22:34:16.470932 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.470911    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.520222 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.520192    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:34:16.542548 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.542519    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88n2\" (UniqueName: \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.542687 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.542566    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.542737 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.542682    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.566991 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.566970    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:34:16.569257 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.569235    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"]
Nov 03 22:34:16.643362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.643333    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88n2\" (UniqueName: \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.643461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.643368    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.643461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.643400    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.643987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.643970    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.645654 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.645622    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.701327 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.701301    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88n2\" (UniqueName: \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\") pod \"maas-api-revoke-sub-7a08b8c9-lrqsq\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:16.783468 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:16.783419    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:17.147533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:17.147497    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"]
Nov 03 22:34:17.148981 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:34:17.148954    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd56813_91fb_453a_8cf9_4f8b41c20b1d.slice/crio-877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf WatchSource:0}: Error finding container 877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf: Status 404 returned error can't find the container with id 877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf
Nov 03 22:34:18.137644 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:18.137613    5533 generic.go:356] "Generic (PLEG): container finished" podID="efd56813-91fb-453a-8cf9-4f8b41c20b1d" containerID="92ef4095272e9928a3b7d931524f610946e2e0e47a61693decbefff02993bcf8" exitCode=0
Nov 03 22:34:18.138014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:18.137704    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq" event={"ID":"efd56813-91fb-453a-8cf9-4f8b41c20b1d","Type":"ContainerDied","Data":"92ef4095272e9928a3b7d931524f610946e2e0e47a61693decbefff02993bcf8"}
Nov 03 22:34:18.138014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:18.137755    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq" event={"ID":"efd56813-91fb-453a-8cf9-4f8b41c20b1d","Type":"ContainerStarted","Data":"877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf"}
Nov 03 22:34:19.261860 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:34:19.261827    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:34:19.269184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.269146    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:19.366887 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.366866    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\" (UniqueName: \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\") pod \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") "
Nov 03 22:34:19.367017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.366901    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\") pod \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") "
Nov 03 22:34:19.367017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.366943    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\") pod \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\" (UID: \"efd56813-91fb-453a-8cf9-4f8b41c20b1d\") "
Nov 03 22:34:19.367287 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.367260    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca" pod "efd56813-91fb-453a-8cf9-4f8b41c20b1d" (UID: "efd56813-91fb-453a-8cf9-4f8b41c20b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:34:19.368980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.368943    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca" pod "efd56813-91fb-453a-8cf9-4f8b41c20b1d" (UID: "efd56813-91fb-453a-8cf9-4f8b41c20b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:34:19.369077 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.368999    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2" pod "efd56813-91fb-453a-8cf9-4f8b41c20b1d" (UID: "efd56813-91fb-453a-8cf9-4f8b41c20b1d"). InnerVolumeSpecName "kube-api-access-z88n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:34:19.468313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.468289    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:34:19.468313 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.468311    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd56813-91fb-453a-8cf9-4f8b41c20b1d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:34:19.468460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:19.468321    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88n2\" (UniqueName: \"kubernetes.io/projected/efd56813-91fb-453a-8cf9-4f8b41c20b1d-kube-api-access-z88n2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:34:20.150252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:20.150218    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq" event={"ID":"efd56813-91fb-453a-8cf9-4f8b41c20b1d","Type":"ContainerDied","Data":"877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf"}
Nov 03 22:34:20.150252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:20.150236    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"
Nov 03 22:34:20.150474 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:20.150304    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877d98f2f4a418e0da29d775c9b9982e0f11ab10f0566d78977a8081f96691cf"
Nov 03 22:34:23.103523 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:23.103491    5533 scope.go:122] "RemoveContainer" containerID="e70002df07c0b5a34ba4ff266e8e9e7b2aabddfdbbebb911477c2ed76d8ecb62"
Nov 03 22:34:23.112192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:23.112174    5533 scope.go:122] "RemoveContainer" containerID="c07d1ecfa4378dcbf108aa3ea58c1a3724b1ed79c380bf824d2e6065218a25a0"
Nov 03 22:34:23.119987 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:23.119967    5533 scope.go:122] "RemoveContainer" containerID="4a393f718d26f0e9297e1dea2449a2960fbf6cf8019317ffedb8aaca7710a199"
Nov 03 22:34:29.196314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:29.196280    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:34:38.109658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:38.108790    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:34:38.109658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:38.108854    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:34:38.149749 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:38.149721    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:34:38.149903 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:38.149777    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:34:40.045728 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:40.045690    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"]
Nov 03 22:34:40.092426 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:40.092392    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c288d3aa-sl94h"]
Nov 03 22:34:40.492099 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:40.492069    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fa7022-0027-4433-864f-fded610f151c" path="/var/lib/kubelet/pods/57fa7022-0027-4433-864f-fded610f151c/volumes"
Nov 03 22:34:41.479555 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:41.479523    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:34:49.260223 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:34:49.260182    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:34:59.208265 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.208184    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"]
Nov 03 22:34:59.208764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.208745    5533 cpu_manager.go:414] "containerMap: removing container" podUID="efd56813-91fb-453a-8cf9-4f8b41c20b1d" containerName="revoke-keys"
Nov 03 22:34:59.208852 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.208766    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd56813-91fb-453a-8cf9-4f8b41c20b1d" containerName="revoke-keys"
Nov 03 22:34:59.208924 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.208910    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd56813-91fb-453a-8cf9-4f8b41c20b1d" containerName="revoke-keys"
Nov 03 22:34:59.212102 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.212080    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.263888 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.263807    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:34:59.264017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.263896    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:34:59.266961 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.266937    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"]
Nov 03 22:34:59.331282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.331257    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgnw\" (UniqueName: \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.331393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.331304    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.331445 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.331412    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.431920 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.431894    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.432055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.431958    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.432055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.432011    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgnw\" (UniqueName: \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.432596 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.432573    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.434303 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.434282    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.489835 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.489784    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgnw\" (UniqueName: \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\") pod \"maas-api-revoke-sub-46064687-hjn2c\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.524561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.524533    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:34:59.888473 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:34:59.888448    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"]
Nov 03 22:34:59.890042 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:34:59.890012    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83154ca_e854_44c2_b366_79f1e62023ab.slice/crio-df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c WatchSource:0}: Error finding container df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c: Status 404 returned error can't find the container with id df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c
Nov 03 22:35:00.396375 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:00.396321    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c" event={"ID":"b83154ca-e854-44c2-b366-79f1e62023ab","Type":"ContainerStarted","Data":"8399956780e341ede97dd1c400255a58f7642ced3e367adb458d10cf1d76b004"}
Nov 03 22:35:00.396813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:00.396417    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c" event={"ID":"b83154ca-e854-44c2-b366-79f1e62023ab","Type":"ContainerStarted","Data":"df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c"}
Nov 03 22:35:00.506420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:00.506371    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c" podStartSLOduration=1.5063586450000002 podStartE2EDuration="1.506358645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:00.505256601 +0000 UTC m=+6402.136570078" watchObservedRunningTime="2026-11-03 22:35:00.506358645 +0000 UTC m=+6402.137672122"
Nov 03 22:35:01.402249 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:01.402216    5533 generic.go:356] "Generic (PLEG): container finished" podID="b83154ca-e854-44c2-b366-79f1e62023ab" containerID="8399956780e341ede97dd1c400255a58f7642ced3e367adb458d10cf1d76b004" exitCode=0
Nov 03 22:35:01.402774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:01.402263    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c" event={"ID":"b83154ca-e854-44c2-b366-79f1e62023ab","Type":"ContainerDied","Data":"8399956780e341ede97dd1c400255a58f7642ced3e367adb458d10cf1d76b004"}
Nov 03 22:35:02.534777 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.534752    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:35:02.659183 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.659108    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\") pod \"b83154ca-e854-44c2-b366-79f1e62023ab\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") "
Nov 03 22:35:02.659316 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.659234    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\" (UniqueName: \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\") pod \"b83154ca-e854-44c2-b366-79f1e62023ab\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") "
Nov 03 22:35:02.659316 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.659270    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\") pod \"b83154ca-e854-44c2-b366-79f1e62023ab\" (UID: \"b83154ca-e854-44c2-b366-79f1e62023ab\") "
Nov 03 22:35:02.661173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.661141    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca" pod "b83154ca-e854-44c2-b366-79f1e62023ab" (UID: "b83154ca-e854-44c2-b366-79f1e62023ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:02.661307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.661232    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca" pod "b83154ca-e854-44c2-b366-79f1e62023ab" (UID: "b83154ca-e854-44c2-b366-79f1e62023ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:35:02.662331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.662302    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw" pod "b83154ca-e854-44c2-b366-79f1e62023ab" (UID: "b83154ca-e854-44c2-b366-79f1e62023ab"). InnerVolumeSpecName "kube-api-access-mqgnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:02.760755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.760732    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqgnw\" (UniqueName: \"kubernetes.io/projected/b83154ca-e854-44c2-b366-79f1e62023ab-kube-api-access-mqgnw\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:02.760755 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.760754    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:02.760898 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:02.760764    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83154ca-e854-44c2-b366-79f1e62023ab-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:03.413851 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:03.413822    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"
Nov 03 22:35:03.414052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:03.413860    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c" event={"ID":"b83154ca-e854-44c2-b366-79f1e62023ab","Type":"ContainerDied","Data":"df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c"}
Nov 03 22:35:03.414052 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:03.413911    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df739f8e449547d955f610b7679857e8a6acb675f9c357e3337598cd86ae663c"
Nov 03 22:35:10.012017 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.011979    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"]
Nov 03 22:35:10.012444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.012386    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b83154ca-e854-44c2-b366-79f1e62023ab" containerName="revoke-keys"
Nov 03 22:35:10.012444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.012398    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83154ca-e854-44c2-b366-79f1e62023ab" containerName="revoke-keys"
Nov 03 22:35:10.012538 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.012469    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83154ca-e854-44c2-b366-79f1e62023ab" containerName="revoke-keys"
Nov 03 22:35:10.015811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.015793    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.068086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.068061    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:35:10.068383 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.068341    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:35:10.071354 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.071324    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"]
Nov 03 22:35:10.123981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.123955    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.124113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.123997    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.124164 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.124138    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjnc\" (UniqueName: \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.225064 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.225036    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjnc\" (UniqueName: \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.225178 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.225073    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.225178 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.225100    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.225761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.225739    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.227376 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.227347    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.282815 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.282759    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjnc\" (UniqueName: \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\") pod \"maas-api-revoke-sub-909045e6-rzmcg\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.330180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.330159    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:10.695189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:10.695165    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"]
Nov 03 22:35:10.697157 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:35:10.697125    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e7808a_fee4_4839_89d3_21574b6185a1.slice/crio-d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55 WatchSource:0}: Error finding container d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55: Status 404 returned error can't find the container with id d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55
Nov 03 22:35:11.456251 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:11.456214    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg" event={"ID":"b2e7808a-fee4-4839-89d3-21574b6185a1","Type":"ContainerStarted","Data":"4a88e9106cfe07768127c55be1bc548ab4589fa167cc176c669fe625e729dd38"}
Nov 03 22:35:11.456892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:11.456277    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg" event={"ID":"b2e7808a-fee4-4839-89d3-21574b6185a1","Type":"ContainerStarted","Data":"d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55"}
Nov 03 22:35:11.567705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:11.567648    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg" podStartSLOduration=2.567614029 podStartE2EDuration="2.567614029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:11.56570246 +0000 UTC m=+6413.197015937" watchObservedRunningTime="2026-11-03 22:35:11.567614029 +0000 UTC m=+6413.198927508"
Nov 03 22:35:12.013668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:12.013625    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:35:12.462779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:12.462747    5533 generic.go:356] "Generic (PLEG): container finished" podID="b2e7808a-fee4-4839-89d3-21574b6185a1" containerID="4a88e9106cfe07768127c55be1bc548ab4589fa167cc176c669fe625e729dd38" exitCode=0
Nov 03 22:35:12.463234 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:12.462838    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg" event={"ID":"b2e7808a-fee4-4839-89d3-21574b6185a1","Type":"ContainerDied","Data":"4a88e9106cfe07768127c55be1bc548ab4589fa167cc176c669fe625e729dd38"}
Nov 03 22:35:13.605497 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.605469    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:13.760192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.760123    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\" (UniqueName: \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\") pod \"b2e7808a-fee4-4839-89d3-21574b6185a1\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") "
Nov 03 22:35:13.760192 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.760169    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\") pod \"b2e7808a-fee4-4839-89d3-21574b6185a1\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") "
Nov 03 22:35:13.760372 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.760252    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\") pod \"b2e7808a-fee4-4839-89d3-21574b6185a1\" (UID: \"b2e7808a-fee4-4839-89d3-21574b6185a1\") "
Nov 03 22:35:13.760621 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.760600    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca" pod "b2e7808a-fee4-4839-89d3-21574b6185a1" (UID: "b2e7808a-fee4-4839-89d3-21574b6185a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:35:13.762311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.762287    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca" pod "b2e7808a-fee4-4839-89d3-21574b6185a1" (UID: "b2e7808a-fee4-4839-89d3-21574b6185a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:35:13.762408 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.762341    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc" pod "b2e7808a-fee4-4839-89d3-21574b6185a1" (UID: "b2e7808a-fee4-4839-89d3-21574b6185a1"). InnerVolumeSpecName "kube-api-access-ppjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:35:13.860901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.860866    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:13.860901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.860898    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjnc\" (UniqueName: \"kubernetes.io/projected/b2e7808a-fee4-4839-89d3-21574b6185a1-kube-api-access-ppjnc\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:13.860901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:13.860908    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2e7808a-fee4-4839-89d3-21574b6185a1-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:35:14.475024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:14.474996    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"
Nov 03 22:35:14.475024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:14.475004    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg" event={"ID":"b2e7808a-fee4-4839-89d3-21574b6185a1","Type":"ContainerDied","Data":"d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55"}
Nov 03 22:35:14.475245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:14.475046    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4fb96d3bc367f4b5c5e48a7e48145a2048929e26cb6ccc8c6bc07cbcdbd4a55"
Nov 03 22:35:19.260842 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:35:19.260810    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:35:22.789135 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:22.789105    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:35:34.988516 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:34.988478    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:35:38.143781 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:38.143739    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:35:38.144168 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:38.143802    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:35:40.044982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:40.044869    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"]
Nov 03 22:35:40.091543 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:40.091511    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mkp49"]
Nov 03 22:35:40.491894 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:40.491861    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac6a63e-9b59-4a43-ae05-296bcd3068f6" path="/var/lib/kubelet/pods/8ac6a63e-9b59-4a43-ae05-296bcd3068f6/volumes"
Nov 03 22:35:47.427393 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:47.427354    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:35:49.329901 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:35:49.329815    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:35:58.092420 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.092388    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"]
Nov 03 22:35:58.092807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.092787    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b2e7808a-fee4-4839-89d3-21574b6185a1" containerName="revoke-keys"
Nov 03 22:35:58.092807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.092799    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e7808a-fee4-4839-89d3-21574b6185a1" containerName="revoke-keys"
Nov 03 22:35:58.092892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.092862    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e7808a-fee4-4839-89d3-21574b6185a1" containerName="revoke-keys"
Nov 03 22:35:58.096216 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.096195    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.150896 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.150869    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"]
Nov 03 22:35:58.194569 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.194546    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:35:58.194710 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.194597    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:35:58.195813 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.195780    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.195933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.195871    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.195933 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.195911    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xb2\" (UniqueName: \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.297117 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.297094    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.297218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.297150    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.297218 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.297172    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xb2\" (UniqueName: \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.297775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.297754    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.299483 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.299463    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.355085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.355029    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xb2\" (UniqueName: \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\") pod \"maas-api-revoke-sub-47430f29-5bcch\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.408862 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.408841    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:35:58.775248 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.775224    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"]
Nov 03 22:35:58.777439 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:35:58.777413    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdccf42_4fce_4b7e_a63d_e7060b2b0863.slice/crio-8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953 WatchSource:0}: Error finding container 8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953: Status 404 returned error can't find the container with id 8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953
Nov 03 22:35:58.789460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:58.789430    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch" event={"ID":"5cdccf42-4fce-4b7e-a63d-e7060b2b0863","Type":"ContainerStarted","Data":"8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953"}
Nov 03 22:35:59.797365 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:59.797300    5533 generic.go:356] "Generic (PLEG): container finished" podID="5cdccf42-4fce-4b7e-a63d-e7060b2b0863" containerID="49fbd918d11710a187731f528a22a150c9404b2f240e2bdd061de18a7b135661" exitCode=0
Nov 03 22:35:59.797705 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:35:59.797389    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch" event={"ID":"5cdccf42-4fce-4b7e-a63d-e7060b2b0863","Type":"ContainerDied","Data":"49fbd918d11710a187731f528a22a150c9404b2f240e2bdd061de18a7b135661"}
Nov 03 22:36:00.932410 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:00.932385    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:36:01.019210 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.019182    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\") pod \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") "
Nov 03 22:36:01.019376 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.019256    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\") pod \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") "
Nov 03 22:36:01.019376 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.019323    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\" (UniqueName: \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\") pod \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\" (UID: \"5cdccf42-4fce-4b7e-a63d-e7060b2b0863\") "
Nov 03 22:36:01.019599 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.019574    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca" pod "5cdccf42-4fce-4b7e-a63d-e7060b2b0863" (UID: "5cdccf42-4fce-4b7e-a63d-e7060b2b0863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:36:01.021305 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.021284    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca" pod "5cdccf42-4fce-4b7e-a63d-e7060b2b0863" (UID: "5cdccf42-4fce-4b7e-a63d-e7060b2b0863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:36:01.021364 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.021349    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2" pod "5cdccf42-4fce-4b7e-a63d-e7060b2b0863" (UID: "5cdccf42-4fce-4b7e-a63d-e7060b2b0863"). InnerVolumeSpecName "kube-api-access-m6xb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:36:01.120497 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.120428    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:01.120497 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.120457    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:01.120497 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.120472    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6xb2\" (UniqueName: \"kubernetes.io/projected/5cdccf42-4fce-4b7e-a63d-e7060b2b0863-kube-api-access-m6xb2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:01.809321 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.809294    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch"
Nov 03 22:36:01.809501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.809342    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47430f29-5bcch" event={"ID":"5cdccf42-4fce-4b7e-a63d-e7060b2b0863","Type":"ContainerDied","Data":"8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953"}
Nov 03 22:36:01.809501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:01.809394    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8034358b73e1783577a40d7b5fa891c9f01585638526a4aa9abd0f5de0f2c953"
Nov 03 22:36:08.897709 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.897676    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"]
Nov 03 22:36:08.898247 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.898227    5533 cpu_manager.go:414] "containerMap: removing container" podUID="5cdccf42-4fce-4b7e-a63d-e7060b2b0863" containerName="revoke-keys"
Nov 03 22:36:08.898346 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.898250    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdccf42-4fce-4b7e-a63d-e7060b2b0863" containerName="revoke-keys"
Nov 03 22:36:08.898408 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.898367    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdccf42-4fce-4b7e-a63d-e7060b2b0863" containerName="revoke-keys"
Nov 03 22:36:08.901929 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.901905    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:08.956580 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.956551    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"]
Nov 03 22:36:08.989207 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.989177    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:08.989355 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.989273    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5rb\" (UniqueName: \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:08.989355 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.989309    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:08.999719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.999696    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:36:08.999826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:08.999755    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:36:09.089901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.089879    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5rb\" (UniqueName: \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.090008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.089906    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.090008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.089966    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.090553 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.090537    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.092224 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.092203    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.147475 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.147451    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5rb\" (UniqueName: \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\") pod \"maas-api-revoke-sub-8a300810-pdmf7\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.218355 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.218328    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:09.586406 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.586377    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"]
Nov 03 22:36:09.588360 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:36:09.588326    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa67633_72d3_49ca_92f1_197777534336.slice/crio-11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b WatchSource:0}: Error finding container 11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b: Status 404 returned error can't find the container with id 11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b
Nov 03 22:36:09.851016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:09.850989    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7" event={"ID":"baa67633-72d3-49ca-92f1-197777534336","Type":"ContainerStarted","Data":"11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b"}
Nov 03 22:36:10.858048 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:10.858012    5533 generic.go:356] "Generic (PLEG): container finished" podID="baa67633-72d3-49ca-92f1-197777534336" containerID="f35b3199583ae4401b1c9ffa5e79abb4fe5c46e8797b620667825766c1fddf07" exitCode=0
Nov 03 22:36:10.858439 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:10.858050    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7" event={"ID":"baa67633-72d3-49ca-92f1-197777534336","Type":"ContainerDied","Data":"f35b3199583ae4401b1c9ffa5e79abb4fe5c46e8797b620667825766c1fddf07"}
Nov 03 22:36:10.867033 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:10.867008    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:36:12.011627 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.011596    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:12.122855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.121952    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\") pod \"baa67633-72d3-49ca-92f1-197777534336\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") "
Nov 03 22:36:12.122855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.122035    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\" (UniqueName: \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\") pod \"baa67633-72d3-49ca-92f1-197777534336\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") "
Nov 03 22:36:12.122855 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.122126    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\") pod \"baa67633-72d3-49ca-92f1-197777534336\" (UID: \"baa67633-72d3-49ca-92f1-197777534336\") "
Nov 03 22:36:12.131668 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.130458    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca" pod "baa67633-72d3-49ca-92f1-197777534336" (UID: "baa67633-72d3-49ca-92f1-197777534336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:36:12.184042 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.184004    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca" pod "baa67633-72d3-49ca-92f1-197777534336" (UID: "baa67633-72d3-49ca-92f1-197777534336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:36:12.235995 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.235959    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa67633-72d3-49ca-92f1-197777534336-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:12.236249 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.236227    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa67633-72d3-49ca-92f1-197777534336-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:12.237136 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.237104    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb" pod "baa67633-72d3-49ca-92f1-197777534336" (UID: "baa67633-72d3-49ca-92f1-197777534336"). InnerVolumeSpecName "kube-api-access-wp5rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:36:12.338129 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.338092    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5rb\" (UniqueName: \"kubernetes.io/projected/baa67633-72d3-49ca-92f1-197777534336-kube-api-access-wp5rb\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:36:12.928970 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.928923    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7" event={"ID":"baa67633-72d3-49ca-92f1-197777534336","Type":"ContainerDied","Data":"11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b"}
Nov 03 22:36:12.929182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.928991    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c781b69aa79788f492aeb910efb00ae1255d4f7cd6c033aa392db1d8f51d1b"
Nov 03 22:36:12.929182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:12.929066    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a300810-pdmf7"
Nov 03 22:36:19.261071 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:36:19.261030    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:36:21.583340 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:21.583309    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:36:23.103269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:23.102752    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:36:23.103269 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:23.102827    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-7lrp6" podUID="6d70fc5c-1dba-4651-b45b-299fa16b1692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:36:23.145558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:23.145435    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:36:23.145558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:23.145513    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:36:23.174240 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:23.174206    5533 scope.go:122] "RemoveContainer" containerID="bbae25d878e3ba9e868daed3f8da53c986ced156e0f53e1f569c06df98e799a0"
Nov 03 22:36:31.189431 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:31.189395    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:36:49.261055 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:36:49.261015    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:36:50.694719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:50.694686    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:36:53.147776 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:53.147738    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-5swkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:36:53.148194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:53.147789    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-5swkz" podUID="25810e8f-c406-4e5b-9194-64d849a8c95e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:36:57.863442 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.863388    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"]
Nov 03 22:36:57.864016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.863991    5533 cpu_manager.go:414] "containerMap: removing container" podUID="baa67633-72d3-49ca-92f1-197777534336" containerName="revoke-keys"
Nov 03 22:36:57.864128 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.864018    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa67633-72d3-49ca-92f1-197777534336" containerName="revoke-keys"
Nov 03 22:36:57.864195 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.864152    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa67633-72d3-49ca-92f1-197777534336" containerName="revoke-keys"
Nov 03 22:36:57.867879 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.867854    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:57.964626 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.964591    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:36:57.964626 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.964620    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:36:57.967196 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:57.967174    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"]
Nov 03 22:36:58.044520 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.044493    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c299v\" (UniqueName: \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.044690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.044570    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.044690 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.044627    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.146020 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.145959    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.146020 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.145992    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.146219 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.146041    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c299v\" (UniqueName: \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.146707 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.146687    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.148348 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.148330    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.203357 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.203333    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c299v\" (UniqueName: \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\") pod \"maas-api-revoke-sub-10ff5642-mpbfq\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.481808 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.481786    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:36:58.846551 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:58.846521    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"]
Nov 03 22:36:58.848794 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:36:58.848764    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f4415e_e795_474e_9563_d32133b00ffa.slice/crio-79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b WatchSource:0}: Error finding container 79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b: Status 404 returned error can't find the container with id 79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b
Nov 03 22:36:59.232360 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:59.232333    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq" event={"ID":"b9f4415e-e795-474e-9563-d32133b00ffa","Type":"ContainerStarted","Data":"40828903cb68f3c2cd38e937dfdd3f7a3befe3203657f0deda766c685984c011"}
Nov 03 22:36:59.232694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:59.232382    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq" event={"ID":"b9f4415e-e795-474e-9563-d32133b00ffa","Type":"ContainerStarted","Data":"79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b"}
Nov 03 22:36:59.341624 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:36:59.341571    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq" podStartSLOduration=2.341554566 podStartE2EDuration="2.341554566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:59.34077806 +0000 UTC m=+6520.972091536" watchObservedRunningTime="2026-11-03 22:36:59.341554566 +0000 UTC m=+6520.972868044"
Nov 03 22:37:00.240012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:00.239982    5533 generic.go:356] "Generic (PLEG): container finished" podID="b9f4415e-e795-474e-9563-d32133b00ffa" containerID="40828903cb68f3c2cd38e937dfdd3f7a3befe3203657f0deda766c685984c011" exitCode=0
Nov 03 22:37:00.240377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:00.240028    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq" event={"ID":"b9f4415e-e795-474e-9563-d32133b00ffa","Type":"ContainerDied","Data":"40828903cb68f3c2cd38e937dfdd3f7a3befe3203657f0deda766c685984c011"}
Nov 03 22:37:01.382592 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.382557    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:37:01.476089 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.476065    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\" (UniqueName: \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\") pod \"b9f4415e-e795-474e-9563-d32133b00ffa\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") "
Nov 03 22:37:01.476235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.476108    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\") pod \"b9f4415e-e795-474e-9563-d32133b00ffa\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") "
Nov 03 22:37:01.476235 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.476142    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\") pod \"b9f4415e-e795-474e-9563-d32133b00ffa\" (UID: \"b9f4415e-e795-474e-9563-d32133b00ffa\") "
Nov 03 22:37:01.476465 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.476443    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca" pod "b9f4415e-e795-474e-9563-d32133b00ffa" (UID: "b9f4415e-e795-474e-9563-d32133b00ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:01.478119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.478096    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca" pod "b9f4415e-e795-474e-9563-d32133b00ffa" (UID: "b9f4415e-e795-474e-9563-d32133b00ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:01.478215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.478122    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v" pod "b9f4415e-e795-474e-9563-d32133b00ffa" (UID: "b9f4415e-e795-474e-9563-d32133b00ffa"). InnerVolumeSpecName "kube-api-access-c299v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:01.577870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.577810    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c299v\" (UniqueName: \"kubernetes.io/projected/b9f4415e-e795-474e-9563-d32133b00ffa-kube-api-access-c299v\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:01.577870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.577835    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:01.577870 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:01.577845    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f4415e-e795-474e-9563-d32133b00ffa-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:02.251950 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:02.251917    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq" event={"ID":"b9f4415e-e795-474e-9563-d32133b00ffa","Type":"ContainerDied","Data":"79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b"}
Nov 03 22:37:02.251950 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:02.251940    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ff5642-mpbfq"
Nov 03 22:37:02.252135 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:02.251970    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c831bf07ea7189dd5b40e2252e3ed6b367aa18de9f14eb9ed6c2405bc8ff7b"
Nov 03 22:37:09.061802 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.061766    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"]
Nov 03 22:37:09.062187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.062170    5533 cpu_manager.go:414] "containerMap: removing container" podUID="b9f4415e-e795-474e-9563-d32133b00ffa" containerName="revoke-keys"
Nov 03 22:37:09.062187 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.062185    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f4415e-e795-474e-9563-d32133b00ffa" containerName="revoke-keys"
Nov 03 22:37:09.062285 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.062262    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f4415e-e795-474e-9563-d32133b00ffa" containerName="revoke-keys"
Nov 03 22:37:09.069868 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.069155    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.121184 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.121152    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"]
Nov 03 22:37:09.132037 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.132011    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.132182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.132052    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpzk\" (UniqueName: \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.132182 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.132075    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.164653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.164612    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:37:09.164771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.164667    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:37:09.233190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.233158    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpzk\" (UniqueName: \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.233347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.233196    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.233347 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.233276    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.234039 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.234015    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.235731 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.235712    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.291043 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.291009    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpzk\" (UniqueName: \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\") pod \"maas-api-revoke-sub-efd89c07-g8kzj\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.384560 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.384491    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:09.756035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:09.756007    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"]
Nov 03 22:37:09.757979 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:37:09.757946    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f6fa8_5410_4878_a500_063ee99e9703.slice/crio-b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44 WatchSource:0}: Error finding container b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44: Status 404 returned error can't find the container with id b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44
Nov 03 22:37:10.294051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:10.294017    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj" event={"ID":"9e6f6fa8-5410-4878-a500-063ee99e9703","Type":"ContainerStarted","Data":"8efd938832a7c1f419ffe0a3f5fb3f7d1d1591bf79466d51320acc0d40d7050d"}
Nov 03 22:37:10.294411 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:10.294072    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj" event={"ID":"9e6f6fa8-5410-4878-a500-063ee99e9703","Type":"ContainerStarted","Data":"b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44"}
Nov 03 22:37:10.405425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:10.405341    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj" podStartSLOduration=1.405328926 podStartE2EDuration="1.405328926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:10.402416993 +0000 UTC m=+6532.033730471" watchObservedRunningTime="2026-11-03 22:37:10.405328926 +0000 UTC m=+6532.036642402"
Nov 03 22:37:11.301715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:11.301686    5533 generic.go:356] "Generic (PLEG): container finished" podID="9e6f6fa8-5410-4878-a500-063ee99e9703" containerID="8efd938832a7c1f419ffe0a3f5fb3f7d1d1591bf79466d51320acc0d40d7050d" exitCode=0
Nov 03 22:37:11.302114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:11.301779    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj" event={"ID":"9e6f6fa8-5410-4878-a500-063ee99e9703","Type":"ContainerDied","Data":"8efd938832a7c1f419ffe0a3f5fb3f7d1d1591bf79466d51320acc0d40d7050d"}
Nov 03 22:37:12.442440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.442416    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:12.468672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.468631    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\") pod \"9e6f6fa8-5410-4878-a500-063ee99e9703\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") "
Nov 03 22:37:12.468820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.468760    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\") pod \"9e6f6fa8-5410-4878-a500-063ee99e9703\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") "
Nov 03 22:37:12.468895 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.468816    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\" (UniqueName: \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\") pod \"9e6f6fa8-5410-4878-a500-063ee99e9703\" (UID: \"9e6f6fa8-5410-4878-a500-063ee99e9703\") "
Nov 03 22:37:12.469046 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.469020    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca" pod "9e6f6fa8-5410-4878-a500-063ee99e9703" (UID: "9e6f6fa8-5410-4878-a500-063ee99e9703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:12.469158 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.469124    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:12.470940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.470912    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca" pod "9e6f6fa8-5410-4878-a500-063ee99e9703" (UID: "9e6f6fa8-5410-4878-a500-063ee99e9703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:12.471116 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.471097    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk" pod "9e6f6fa8-5410-4878-a500-063ee99e9703" (UID: "9e6f6fa8-5410-4878-a500-063ee99e9703"). InnerVolumeSpecName "kube-api-access-kkpzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:12.471607 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.471587    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:37:12.569688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.569613    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f6fa8-5410-4878-a500-063ee99e9703-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:12.569688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:12.569654    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpzk\" (UniqueName: \"kubernetes.io/projected/9e6f6fa8-5410-4878-a500-063ee99e9703-kube-api-access-kkpzk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:13.315282 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:13.315244    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj"
Nov 03 22:37:13.315460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:13.315291    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd89c07-g8kzj" event={"ID":"9e6f6fa8-5410-4878-a500-063ee99e9703","Type":"ContainerDied","Data":"b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44"}
Nov 03 22:37:13.315460 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:13.315352    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d6402e92dacf5362c33459124e9ba7f95b40c72bdc7749386ed770cced8d44"
Nov 03 22:37:19.259662 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:37:19.259604    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:37:19.812845 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.812811    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"]
Nov 03 22:37:19.813223 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.813208    5533 cpu_manager.go:414] "containerMap: removing container" podUID="9e6f6fa8-5410-4878-a500-063ee99e9703" containerName="revoke-keys"
Nov 03 22:37:19.813223 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.813223    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6f6fa8-5410-4878-a500-063ee99e9703" containerName="revoke-keys"
Nov 03 22:37:19.813359 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.813314    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6f6fa8-5410-4878-a500-063ee99e9703" containerName="revoke-keys"
Nov 03 22:37:19.816524 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.816502    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.839206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.839174    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.839362 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.839227    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.839436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.839382    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6vf\" (UniqueName: \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.872423 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.872389    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"]
Nov 03 22:37:19.915021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.914993    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:37:19.915170 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.915027    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:37:19.940215 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.940187    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6vf\" (UniqueName: \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.940337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.940252    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.940337 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.940283    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.941075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.941054    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.942780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.942756    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:19.998070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:19.998039    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6vf\" (UniqueName: \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\") pod \"maas-api-revoke-sub-067b78fe-rzgcq\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:20.129594 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:20.129534    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:20.501286 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:20.501260    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"]
Nov 03 22:37:20.503264 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:37:20.503223    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4c1f97_2c9b_4750_91e5_a20653fb7fe5.slice/crio-4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7 WatchSource:0}: Error finding container 4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7: Status 404 returned error can't find the container with id 4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7
Nov 03 22:37:21.369685 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:21.369631    5533 generic.go:356] "Generic (PLEG): container finished" podID="1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" containerID="df85e7b3bd05eda6ef4744801560fbbbddbbe2d124b6d218b4d7529b4beb561f" exitCode=0
Nov 03 22:37:21.369823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:21.369708    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq" event={"ID":"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5","Type":"ContainerDied","Data":"df85e7b3bd05eda6ef4744801560fbbbddbbe2d124b6d218b4d7529b4beb561f"}
Nov 03 22:37:21.369823 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:21.369767    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq" event={"ID":"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5","Type":"ContainerStarted","Data":"4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7"}
Nov 03 22:37:22.497508 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.497478    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:22.560740 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.560715    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\") pod \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") "
Nov 03 22:37:22.560843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.560780    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\" (UniqueName: \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\") pod \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") "
Nov 03 22:37:22.560843 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.560820    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\") pod \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\" (UID: \"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5\") "
Nov 03 22:37:22.561063 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.561041    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca" pod "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" (UID: "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:22.562792 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.562769    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca" pod "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" (UID: "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:22.562892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.562872    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf" pod "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" (UID: "1a4c1f97-2c9b-4750-91e5-a20653fb7fe5"). InnerVolumeSpecName "kube-api-access-7r6vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:22.661484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.661434    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:22.661484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.661455    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:22.661484 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:22.661465    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6vf\" (UniqueName: \"kubernetes.io/projected/1a4c1f97-2c9b-4750-91e5-a20653fb7fe5-kube-api-access-7r6vf\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:23.274670 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:23.274620    5533 scope.go:122] "RemoveContainer" containerID="f38791331d2327c76c02cfe38ff73b3df23b3bd2a5bd55c1dca8b7b759d5cfae"
Nov 03 22:37:23.380916 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:23.380891    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq"
Nov 03 22:37:23.381002 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:23.380916    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067b78fe-rzgcq" event={"ID":"1a4c1f97-2c9b-4750-91e5-a20653fb7fe5","Type":"ContainerDied","Data":"4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7"}
Nov 03 22:37:23.381002 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:23.380983    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0265197585cfa10989da31862c5e3585c66aa0614d9d8b625fb7a4bbc8c5b7"
Nov 03 22:37:31.044616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.044579    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"]
Nov 03 22:37:31.045169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.045147    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" containerName="revoke-keys"
Nov 03 22:37:31.045270 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.045172    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" containerName="revoke-keys"
Nov 03 22:37:31.045331 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.045283    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4c1f97-2c9b-4750-91e5-a20653fb7fe5" containerName="revoke-keys"
Nov 03 22:37:31.050012 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.049988    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.100462 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.100417    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:37:31.104438 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.104407    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"]
Nov 03 22:37:31.125826 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.125792    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.125967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.125898    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.125967 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.125937    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx7r\" (UniqueName: \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.147085 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.147051    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:37:31.226803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.226777    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.226946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.226822    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.226946 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.226941    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx7r\" (UniqueName: \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.227506 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.227473    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.229101 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.229082    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.284986 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.284957    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx7r\" (UniqueName: \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\") pod \"maas-api-revoke-sub-76a5d921-jcmd8\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.363223 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.363167    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:31.735189 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:31.735156    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"]
Nov 03 22:37:31.736387 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:37:31.736360    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0858251b_00c9_4d10_a085_535a84feee15.slice/crio-eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9 WatchSource:0}: Error finding container eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9: Status 404 returned error can't find the container with id eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9
Nov 03 22:37:32.434499 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:32.434466    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8" event={"ID":"0858251b-00c9-4d10-a085-535a84feee15","Type":"ContainerStarted","Data":"82b823d978219c3bb7689b192eea1c1b2b325c37eb781d89871471ef7ce2cb33"}
Nov 03 22:37:32.434905 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:32.434530    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8" event={"ID":"0858251b-00c9-4d10-a085-535a84feee15","Type":"ContainerStarted","Data":"eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9"}
Nov 03 22:37:32.544096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:32.544054    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8" podStartSLOduration=1.544042063 podStartE2EDuration="1.544042063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:32.54286279 +0000 UTC m=+6554.174176282" watchObservedRunningTime="2026-11-03 22:37:32.544042063 +0000 UTC m=+6554.175355539"
Nov 03 22:37:33.441879 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:33.441845    5533 generic.go:356] "Generic (PLEG): container finished" podID="0858251b-00c9-4d10-a085-535a84feee15" containerID="82b823d978219c3bb7689b192eea1c1b2b325c37eb781d89871471ef7ce2cb33" exitCode=0
Nov 03 22:37:33.442330 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:33.441937    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8" event={"ID":"0858251b-00c9-4d10-a085-535a84feee15","Type":"ContainerDied","Data":"82b823d978219c3bb7689b192eea1c1b2b325c37eb781d89871471ef7ce2cb33"}
Nov 03 22:37:34.604780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.604745    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:34.657521 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.657491    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\") pod \"0858251b-00c9-4d10-a085-535a84feee15\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") "
Nov 03 22:37:34.657686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.657527    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\" (UniqueName: \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\") pod \"0858251b-00c9-4d10-a085-535a84feee15\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") "
Nov 03 22:37:34.657686 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.657553    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\") pod \"0858251b-00c9-4d10-a085-535a84feee15\" (UID: \"0858251b-00c9-4d10-a085-535a84feee15\") "
Nov 03 22:37:34.657984 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.657944    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca" pod "0858251b-00c9-4d10-a085-535a84feee15" (UID: "0858251b-00c9-4d10-a085-535a84feee15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:34.659617 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.659595    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r" pod "0858251b-00c9-4d10-a085-535a84feee15" (UID: "0858251b-00c9-4d10-a085-535a84feee15"). InnerVolumeSpecName "kube-api-access-jbx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:34.659737 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.659683    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca" pod "0858251b-00c9-4d10-a085-535a84feee15" (UID: "0858251b-00c9-4d10-a085-535a84feee15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:34.758550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.758463    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0858251b-00c9-4d10-a085-535a84feee15-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:34.758550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.758495    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbx7r\" (UniqueName: \"kubernetes.io/projected/0858251b-00c9-4d10-a085-535a84feee15-kube-api-access-jbx7r\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:34.758550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:34.758506    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0858251b-00c9-4d10-a085-535a84feee15-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:35.465122 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:35.465086    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8"
Nov 03 22:37:35.465336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:35.465121    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76a5d921-jcmd8" event={"ID":"0858251b-00c9-4d10-a085-535a84feee15","Type":"ContainerDied","Data":"eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9"}
Nov 03 22:37:35.465336 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:35.465171    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaba7695ec91d93d0b3fdd986cd2cb26b9d0d1370e23db11a4a168fb83da41e9"
Nov 03 22:37:37.048008 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:37.047975    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"]
Nov 03 22:37:37.094908 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:37.094855    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hh9f4"]
Nov 03 22:37:38.492585 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:38.492538    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cc89da-041b-47cf-addb-4febb0fd9b0e" path="/var/lib/kubelet/pods/61cc89da-041b-47cf-addb-4febb0fd9b0e/volumes"
Nov 03 22:37:41.752471 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.752441    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"]
Nov 03 22:37:41.752889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.752829    5533 cpu_manager.go:414] "containerMap: removing container" podUID="0858251b-00c9-4d10-a085-535a84feee15" containerName="revoke-keys"
Nov 03 22:37:41.752889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.752845    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0858251b-00c9-4d10-a085-535a84feee15" containerName="revoke-keys"
Nov 03 22:37:41.752980 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.752928    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="0858251b-00c9-4d10-a085-535a84feee15" containerName="revoke-keys"
Nov 03 22:37:41.756361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.756341    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.808684 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.808631    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:37:41.811726 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.811698    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"]
Nov 03 22:37:41.854070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.854044    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.854177 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.854088    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbmk\" (UniqueName: \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.854177 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.854112    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.855211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.855191    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:37:41.955419 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.955385    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.955529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.955438    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbmk\" (UniqueName: \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.955529 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.955463    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.956090 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.956068    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:41.957754 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:41.957734    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:42.012830 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:42.012779    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbmk\" (UniqueName: \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\") pod \"maas-api-revoke-sub-f39f0c36-ng989\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:42.068839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:42.068816    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:42.438245 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:42.438218    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"]
Nov 03 22:37:42.440169 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:37:42.440141    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86949266_8b11_4d28_ab2b_0c1e1ea6ccde.slice/crio-346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739 WatchSource:0}: Error finding container 346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739: Status 404 returned error can't find the container with id 346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739
Nov 03 22:37:42.550547 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:42.550523    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989" event={"ID":"86949266-8b11-4d28-ab2b-0c1e1ea6ccde","Type":"ContainerStarted","Data":"346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739"}
Nov 03 22:37:43.557820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:43.557750    5533 generic.go:356] "Generic (PLEG): container finished" podID="86949266-8b11-4d28-ab2b-0c1e1ea6ccde" containerID="ed4831728cbc12552a7a810a9f2a9ec38ed66f8ba4fefb794cf9c2e23761f603" exitCode=0
Nov 03 22:37:43.557820 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:43.557789    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989" event={"ID":"86949266-8b11-4d28-ab2b-0c1e1ea6ccde","Type":"ContainerDied","Data":"ed4831728cbc12552a7a810a9f2a9ec38ed66f8ba4fefb794cf9c2e23761f603"}
Nov 03 22:37:44.694518 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.694493    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:44.779327 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.779298    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\" (UniqueName: \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\") pod \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") "
Nov 03 22:37:44.779444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.779336    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\") pod \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") "
Nov 03 22:37:44.779444 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.779375    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\") pod \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\" (UID: \"86949266-8b11-4d28-ab2b-0c1e1ea6ccde\") "
Nov 03 22:37:44.779796 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.779769    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca" pod "86949266-8b11-4d28-ab2b-0c1e1ea6ccde" (UID: "86949266-8b11-4d28-ab2b-0c1e1ea6ccde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:44.781385 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.781360    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca" pod "86949266-8b11-4d28-ab2b-0c1e1ea6ccde" (UID: "86949266-8b11-4d28-ab2b-0c1e1ea6ccde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:44.781480 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.781367    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk" pod "86949266-8b11-4d28-ab2b-0c1e1ea6ccde" (UID: "86949266-8b11-4d28-ab2b-0c1e1ea6ccde"). InnerVolumeSpecName "kube-api-access-mnbmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:44.880938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.880885    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbmk\" (UniqueName: \"kubernetes.io/projected/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-kube-api-access-mnbmk\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:44.880938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.880910    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:44.880938 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:44.880920    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86949266-8b11-4d28-ab2b-0c1e1ea6ccde-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:45.576109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:45.576077    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989"
Nov 03 22:37:45.576267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:45.576122    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39f0c36-ng989" event={"ID":"86949266-8b11-4d28-ab2b-0c1e1ea6ccde","Type":"ContainerDied","Data":"346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739"}
Nov 03 22:37:45.576267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:45.576174    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346035ea3f4573f840a4866862cefb9daa406a44ae1c313b0aefd066c0900739"
Nov 03 22:37:49.259524 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:37:49.259493    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:37:52.739022 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.738984    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"]
Nov 03 22:37:52.739550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.739392    5533 cpu_manager.go:414] "containerMap: removing container" podUID="86949266-8b11-4d28-ab2b-0c1e1ea6ccde" containerName="revoke-keys"
Nov 03 22:37:52.739550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.739404    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86949266-8b11-4d28-ab2b-0c1e1ea6ccde" containerName="revoke-keys"
Nov 03 22:37:52.739550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.739470    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="86949266-8b11-4d28-ab2b-0c1e1ea6ccde" containerName="revoke-keys"
Nov 03 22:37:52.742715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.742694    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.794558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.794526    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:37:52.798654 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.798610    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"]
Nov 03 22:37:52.841021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.840991    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:37:52.843954 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.843922    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zx7\" (UniqueName: \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.844088 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.843964    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.844154 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.844084    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.945495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.945458    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zx7\" (UniqueName: \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.945688 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.945519    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.945766 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.945674    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.946301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.946274    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:52.948045 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:52.948023    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:53.003982 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:53.003901    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zx7\" (UniqueName: \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\") pod \"maas-api-revoke-sub-95b538da-tvgxd\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:53.057683 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:53.057629    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:53.432761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:53.432734    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"]
Nov 03 22:37:53.434449 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:37:53.434422    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6811be6_7532_4fff_86a2_ee0ef5d94c98.slice/crio-968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec WatchSource:0}: Error finding container 968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec: Status 404 returned error can't find the container with id 968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec
Nov 03 22:37:53.621425 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:53.621396    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd" event={"ID":"a6811be6-7532-4fff-86a2-ee0ef5d94c98","Type":"ContainerStarted","Data":"968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec"}
Nov 03 22:37:54.627889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:54.627813    5533 generic.go:356] "Generic (PLEG): container finished" podID="a6811be6-7532-4fff-86a2-ee0ef5d94c98" containerID="c2ddf76157513786453ea9607db828c2c6f85c2a318569d1bf4acc778520bb40" exitCode=0
Nov 03 22:37:54.628301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:54.627903    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd" event={"ID":"a6811be6-7532-4fff-86a2-ee0ef5d94c98","Type":"ContainerDied","Data":"c2ddf76157513786453ea9607db828c2c6f85c2a318569d1bf4acc778520bb40"}
Nov 03 22:37:55.767527 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.767503    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:55.873412 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.873387    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\") pod \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") "
Nov 03 22:37:55.873564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.873468    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\" (UniqueName: \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\") pod \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") "
Nov 03 22:37:55.873564 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.873548    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\") pod \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\" (UID: \"a6811be6-7532-4fff-86a2-ee0ef5d94c98\") "
Nov 03 22:37:55.874004 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.873974    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca" pod "a6811be6-7532-4fff-86a2-ee0ef5d94c98" (UID: "a6811be6-7532-4fff-86a2-ee0ef5d94c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:37:55.875715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.875691    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7" pod "a6811be6-7532-4fff-86a2-ee0ef5d94c98" (UID: "a6811be6-7532-4fff-86a2-ee0ef5d94c98"). InnerVolumeSpecName "kube-api-access-j8zx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:37:55.875822 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.875718    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca" pod "a6811be6-7532-4fff-86a2-ee0ef5d94c98" (UID: "a6811be6-7532-4fff-86a2-ee0ef5d94c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:37:55.974416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.974387    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zx7\" (UniqueName: \"kubernetes.io/projected/a6811be6-7532-4fff-86a2-ee0ef5d94c98-kube-api-access-j8zx7\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:55.974416 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.974415    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:55.974576 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:55.974427    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6811be6-7532-4fff-86a2-ee0ef5d94c98-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:37:56.640504 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:56.640479    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd"
Nov 03 22:37:56.640709 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:56.640474    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b538da-tvgxd" event={"ID":"a6811be6-7532-4fff-86a2-ee0ef5d94c98","Type":"ContainerDied","Data":"968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec"}
Nov 03 22:37:56.640709 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:56.640617    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="968726dfb3ee68b11e8213e571efda99892093bc850aad5d02f6eed3d0e42fec"
Nov 03 22:37:58.046984 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:58.046945    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"]
Nov 03 22:37:58.106679 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:58.098893    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2bwms"]
Nov 03 22:37:58.520238 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:37:58.520201    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fc0495-071c-499e-907c-f064006b5cc5" path="/var/lib/kubelet/pods/c8fc0495-071c-499e-907c-f064006b5cc5/volumes"
Nov 03 22:38:03.456115 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.456034    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"]
Nov 03 22:38:03.456603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.456428    5533 cpu_manager.go:414] "containerMap: removing container" podUID="a6811be6-7532-4fff-86a2-ee0ef5d94c98" containerName="revoke-keys"
Nov 03 22:38:03.456603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.456440    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6811be6-7532-4fff-86a2-ee0ef5d94c98" containerName="revoke-keys"
Nov 03 22:38:03.456603 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.456505    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6811be6-7532-4fff-86a2-ee0ef5d94c98" containerName="revoke-keys"
Nov 03 22:38:03.459983 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.459965    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.466514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.466230    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.466514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.466365    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzrq\" (UniqueName: \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.466514 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.466442    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.558496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.558474    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:03.558624 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.558603    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:03.561047 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.561026    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"]
Nov 03 22:38:03.566959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.566941    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.567025 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.566978    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.567070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.567052    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzrq\" (UniqueName: \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.567644 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.567613    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.569370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.569352    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.624590 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.624571    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzrq\" (UniqueName: \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\") pod \"maas-api-revoke-sub-b37834e6-hp2q5\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:03.772779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:03.772720    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:04.140981 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:04.140957    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"]
Nov 03 22:38:04.142757 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:04.142731    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe79f06a_aedc_4093_9e1c_a78d01d0bd54.slice/crio-815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7 WatchSource:0}: Error finding container 815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7: Status 404 returned error can't find the container with id 815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7
Nov 03 22:38:04.719901 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:04.719854    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5" event={"ID":"be79f06a-aedc-4093-9e1c-a78d01d0bd54","Type":"ContainerStarted","Data":"d37b28a6d7291805e88577b17460bd94418b2410a253b129f8b12fdaa635fb7f"}
Nov 03 22:38:04.720274 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:04.719917    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5" event={"ID":"be79f06a-aedc-4093-9e1c-a78d01d0bd54","Type":"ContainerStarted","Data":"815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7"}
Nov 03 22:38:04.831496 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:04.831443    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5" podStartSLOduration=1.83143083 podStartE2EDuration="1.83143083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:04.829073032 +0000 UTC m=+6586.460386506" watchObservedRunningTime="2026-11-03 22:38:04.83143083 +0000 UTC m=+6586.462744307"
Nov 03 22:38:05.726986 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:05.726951    5533 generic.go:356] "Generic (PLEG): container finished" podID="be79f06a-aedc-4093-9e1c-a78d01d0bd54" containerID="d37b28a6d7291805e88577b17460bd94418b2410a253b129f8b12fdaa635fb7f" exitCode=0
Nov 03 22:38:05.727361 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:05.727033    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5" event={"ID":"be79f06a-aedc-4093-9e1c-a78d01d0bd54","Type":"ContainerDied","Data":"d37b28a6d7291805e88577b17460bd94418b2410a253b129f8b12fdaa635fb7f"}
Nov 03 22:38:06.865280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.865253    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:06.893582 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.893561    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\") pod \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") "
Nov 03 22:38:06.893724 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.893612    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\" (UniqueName: \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\") pod \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") "
Nov 03 22:38:06.893724 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.893672    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\") pod \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\" (UID: \"be79f06a-aedc-4093-9e1c-a78d01d0bd54\") "
Nov 03 22:38:06.893919 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.893895    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca" pod "be79f06a-aedc-4093-9e1c-a78d01d0bd54" (UID: "be79f06a-aedc-4093-9e1c-a78d01d0bd54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:06.895678 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.895657    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca" pod "be79f06a-aedc-4093-9e1c-a78d01d0bd54" (UID: "be79f06a-aedc-4093-9e1c-a78d01d0bd54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:06.895753 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.895673    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq" pod "be79f06a-aedc-4093-9e1c-a78d01d0bd54" (UID: "be79f06a-aedc-4093-9e1c-a78d01d0bd54"). InnerVolumeSpecName "kube-api-access-sdzrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:06.994628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.994561    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:06.994628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.994583    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzrq\" (UniqueName: \"kubernetes.io/projected/be79f06a-aedc-4093-9e1c-a78d01d0bd54-kube-api-access-sdzrq\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:06.994628 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:06.994593    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be79f06a-aedc-4093-9e1c-a78d01d0bd54-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:07.739661 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:07.739616    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5"
Nov 03 22:38:07.739839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:07.739670    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37834e6-hp2q5" event={"ID":"be79f06a-aedc-4093-9e1c-a78d01d0bd54","Type":"ContainerDied","Data":"815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7"}
Nov 03 22:38:07.739839 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:07.739719    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815405c031c76e49c8edc188840f5265953e017d07b536418f6b2ff92d93d3e7"
Nov 03 22:38:14.214293 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.214258    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"]
Nov 03 22:38:14.214774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.214657    5533 cpu_manager.go:414] "containerMap: removing container" podUID="be79f06a-aedc-4093-9e1c-a78d01d0bd54" containerName="revoke-keys"
Nov 03 22:38:14.214774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.214669    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be79f06a-aedc-4093-9e1c-a78d01d0bd54" containerName="revoke-keys"
Nov 03 22:38:14.214774 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.214735    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="be79f06a-aedc-4093-9e1c-a78d01d0bd54" containerName="revoke-keys"
Nov 03 22:38:14.217749 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.217725    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.254545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.254362    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.254545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.254417    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.254545 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.254450    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g88f\" (UniqueName: \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.277664 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.273039    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"]
Nov 03 22:38:14.316096 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.316074    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:14.316236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.316162    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:14.355390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.355362    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g88f\" (UniqueName: \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.355533 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.355514    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.355584 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.355549    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.356170 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.356149    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.357918 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.357896    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.413307 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.413285    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g88f\" (UniqueName: \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\") pod \"maas-api-revoke-sub-7b44d61d-5gmmx\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.530451 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.530389    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:14.895382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:14.895298    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"]
Nov 03 22:38:14.897780 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:14.897753    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00b4e74_19c8_4bf4_b0a5_c9e9a7dfea6e.slice/crio-6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a WatchSource:0}: Error finding container 6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a: Status 404 returned error can't find the container with id 6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a
Nov 03 22:38:15.792575 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:15.792536    5533 generic.go:356] "Generic (PLEG): container finished" podID="e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" containerID="e61aaf9955f8e52e682d87908515924f62c52644a74282235a254fb5fbb5413b" exitCode=0
Nov 03 22:38:15.792976 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:15.792572    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx" event={"ID":"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e","Type":"ContainerDied","Data":"e61aaf9955f8e52e682d87908515924f62c52644a74282235a254fb5fbb5413b"}
Nov 03 22:38:15.792976 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:15.792691    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx" event={"ID":"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e","Type":"ContainerStarted","Data":"6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a"}
Nov 03 22:38:16.973005 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:16.972978    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:17.080503 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.080428    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\") pod \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") "
Nov 03 22:38:17.080503 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.080482    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\" (UniqueName: \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\") pod \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") "
Nov 03 22:38:17.080760 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.080594    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\") pod \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\" (UID: \"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e\") "
Nov 03 22:38:17.080925 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.080897    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca" pod "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" (UID: "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:17.086928 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.086893    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca" pod "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" (UID: "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:17.087153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.087134    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f" pod "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" (UID: "e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e"). InnerVolumeSpecName "kube-api-access-8g88f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:17.181680 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.181624    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:17.181680 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.181682    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:17.181962 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.181701    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g88f\" (UniqueName: \"kubernetes.io/projected/e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e-kube-api-access-8g88f\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:17.824035 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.823997    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx" event={"ID":"e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e","Type":"ContainerDied","Data":"6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a"}
Nov 03 22:38:17.824319 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.824296    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5e3415a21697ca0345602112643ac7d7150cc4de7dbaca3f0f10c567575f6a"
Nov 03 22:38:17.824513 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:17.824496    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b44d61d-5gmmx"
Nov 03 22:38:19.273743 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:38:19.273711    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:38:19.650476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:19.650403    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:38:19.662213 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:19.662192    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:38:19.705075 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:19.705052    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a764rf_83bcf0cc-19d3-4896-b41a-d5e8eee25017/extract/0.log"
Nov 03 22:38:19.734153 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:19.734131    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n99n6_c8fc55c3-89a7-4d55-bae4-a64d15efebaa/ovn-acl-logging/0.log"
Nov 03 22:38:25.033658 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.033611    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"]
Nov 03 22:38:25.034059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.034019    5533 cpu_manager.go:414] "containerMap: removing container" podUID="e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" containerName="revoke-keys"
Nov 03 22:38:25.034059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.034034    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" containerName="revoke-keys"
Nov 03 22:38:25.034142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.034109    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e" containerName="revoke-keys"
Nov 03 22:38:25.037646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.037620    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.084308 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.084277    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.084422 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.084321    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlc5t\" (UniqueName: \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.084483 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.084413    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.088751 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.088726    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:25.092301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.092278    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"]
Nov 03 22:38:25.135435 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.135414    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:25.185262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.185237    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.185387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.185276    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlc5t\" (UniqueName: \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.185387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.185318    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.185964 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.185943    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.187387 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.187370    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.242611 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.242588    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlc5t\" (UniqueName: \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\") pod \"maas-api-revoke-sub-9a476202-f6khf\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.350206 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.350152    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:25.714142 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.714116    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"]
Nov 03 22:38:25.716108 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:25.716078    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4bf38e_66dc_4a27_bcbe_dcd84c9850a2.slice/crio-7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd WatchSource:0}: Error finding container 7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd: Status 404 returned error can't find the container with id 7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd
Nov 03 22:38:25.884694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:25.884660    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf" event={"ID":"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2","Type":"ContainerStarted","Data":"7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd"}
Nov 03 22:38:26.900814 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:26.900730    5533 generic.go:356] "Generic (PLEG): container finished" podID="1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" containerID="64e9c88c3ba1ade4c240174f4e3cd07e5b905fe9bffb3092f31db603291bbfbc" exitCode=0
Nov 03 22:38:26.900814 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:26.900790    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf" event={"ID":"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2","Type":"ContainerDied","Data":"64e9c88c3ba1ade4c240174f4e3cd07e5b905fe9bffb3092f31db603291bbfbc"}
Nov 03 22:38:28.052016 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.051994    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:28.112118 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.112081    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\") pod \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") "
Nov 03 22:38:28.112291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.112152    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\" (UniqueName: \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\") pod \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") "
Nov 03 22:38:28.112291 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.112235    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\") pod \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\" (UID: \"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2\") "
Nov 03 22:38:28.112734 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.112705    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca" pod "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" (UID: "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:28.114650 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.114606    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t" pod "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" (UID: "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2"). InnerVolumeSpecName "kube-api-access-vlc5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:28.114768 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.114680    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca" pod "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" (UID: "1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:28.213694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.213663    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:28.213694 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.213691    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlc5t\" (UniqueName: \"kubernetes.io/projected/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-kube-api-access-vlc5t\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:28.213872 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.213706    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:28.924917 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.924890    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf"
Nov 03 22:38:28.925464 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.925429    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a476202-f6khf" event={"ID":"1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2","Type":"ContainerDied","Data":"7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd"}
Nov 03 22:38:28.925699 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:28.925681    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e777dd945281f61b293afa9ed4eaa7b2adf90856c913d07e6da3f9e941a97fd"
Nov 03 22:38:35.805884 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.805852    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"]
Nov 03 22:38:35.806278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.806235    5533 cpu_manager.go:414] "containerMap: removing container" podUID="1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" containerName="revoke-keys"
Nov 03 22:38:35.806278 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.806246    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" containerName="revoke-keys"
Nov 03 22:38:35.806353 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.806304    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2" containerName="revoke-keys"
Nov 03 22:38:35.809550 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.809528    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.864879 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.864847    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"]
Nov 03 22:38:35.892315 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.892287    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.892494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.892422    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpkm\" (UniqueName: \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.892494 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.892476    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.908014 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.907990    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:35.908118 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.908047    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:35.993662 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.993619    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpkm\" (UniqueName: \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.993788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.993681    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.993788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.993750    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.994314 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.994296    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:35.996122 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:35.996097    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:36.046676 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.046649    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"]
Nov 03 22:38:36.052158 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.052135    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03a9a92-5kwld"]
Nov 03 22:38:36.052366 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.052347    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpkm\" (UniqueName: \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\") pod \"maas-api-revoke-sub-b2a22b3e-dm8dr\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:36.122965 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.122915    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:36.497667 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:36.493077    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73eeab38_1902_4c50_9763_2bc0ba9c9829.slice/crio-ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b WatchSource:0}: Error finding container ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b: Status 404 returned error can't find the container with id ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b
Nov 03 22:38:36.497667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.494817    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5816f8-a46c-4050-9a11-97de66b7f911" path="/var/lib/kubelet/pods/5e5816f8-a46c-4050-9a11-97de66b7f911/volumes"
Nov 03 22:38:36.497667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.495224    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"]
Nov 03 22:38:36.976945 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.976908    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr" event={"ID":"73eeab38-1902-4c50-9763-2bc0ba9c9829","Type":"ContainerStarted","Data":"70bd977c5677c5f7d046fe10e208f86d46bdcbdc6f14afed69b5b1180b51a4ae"}
Nov 03 22:38:36.977304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:36.976973    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr" event={"ID":"73eeab38-1902-4c50-9763-2bc0ba9c9829","Type":"ContainerStarted","Data":"ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b"}
Nov 03 22:38:37.087863 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:37.087768    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr" podStartSLOduration=2.087753718 podStartE2EDuration="2.087753718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:37.085143704 +0000 UTC m=+6618.716457181" watchObservedRunningTime="2026-11-03 22:38:37.087753718 +0000 UTC m=+6618.719067194"
Nov 03 22:38:37.983775 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:37.983741    5533 generic.go:356] "Generic (PLEG): container finished" podID="73eeab38-1902-4c50-9763-2bc0ba9c9829" containerID="70bd977c5677c5f7d046fe10e208f86d46bdcbdc6f14afed69b5b1180b51a4ae" exitCode=0
Nov 03 22:38:37.984179 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:37.983835    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr" event={"ID":"73eeab38-1902-4c50-9763-2bc0ba9c9829","Type":"ContainerDied","Data":"70bd977c5677c5f7d046fe10e208f86d46bdcbdc6f14afed69b5b1180b51a4ae"}
Nov 03 22:38:39.135304 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.135282    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:39.222501 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.222474    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\") pod \"73eeab38-1902-4c50-9763-2bc0ba9c9829\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") "
Nov 03 22:38:39.222691 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.222513    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\" (UniqueName: \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\") pod \"73eeab38-1902-4c50-9763-2bc0ba9c9829\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") "
Nov 03 22:38:39.222691 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.222547    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\") pod \"73eeab38-1902-4c50-9763-2bc0ba9c9829\" (UID: \"73eeab38-1902-4c50-9763-2bc0ba9c9829\") "
Nov 03 22:38:39.222975 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.222947    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca" pod "73eeab38-1902-4c50-9763-2bc0ba9c9829" (UID: "73eeab38-1902-4c50-9763-2bc0ba9c9829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:39.224702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.224678    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm" pod "73eeab38-1902-4c50-9763-2bc0ba9c9829" (UID: "73eeab38-1902-4c50-9763-2bc0ba9c9829"). InnerVolumeSpecName "kube-api-access-2hpkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:39.225180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.225155    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca" pod "73eeab38-1902-4c50-9763-2bc0ba9c9829" (UID: "73eeab38-1902-4c50-9763-2bc0ba9c9829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:39.325711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.325172    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:39.325711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.325213    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hpkm\" (UniqueName: \"kubernetes.io/projected/73eeab38-1902-4c50-9763-2bc0ba9c9829-kube-api-access-2hpkm\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:39.325711 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:39.325230    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eeab38-1902-4c50-9763-2bc0ba9c9829-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:40.014656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:40.013899    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr" event={"ID":"73eeab38-1902-4c50-9763-2bc0ba9c9829","Type":"ContainerDied","Data":"ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b"}
Nov 03 22:38:40.014656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:40.013984    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace3ef0690fa6a7e6fb746af412e2eadf674d7ad8a0e469b089d57752f253d4b"
Nov 03 22:38:40.014656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:40.014076    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a22b3e-dm8dr"
Nov 03 22:38:40.677788 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:40.677731    5533 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 22:38:40.678424 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:40.678375    5533 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-z2qhq" podUID="fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 22:38:46.570554 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.570513    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"]
Nov 03 22:38:46.571130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.570975    5533 cpu_manager.go:414] "containerMap: removing container" podUID="73eeab38-1902-4c50-9763-2bc0ba9c9829" containerName="revoke-keys"
Nov 03 22:38:46.571130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.570990    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73eeab38-1902-4c50-9763-2bc0ba9c9829" containerName="revoke-keys"
Nov 03 22:38:46.571130 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.571095    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="73eeab38-1902-4c50-9763-2bc0ba9c9829" containerName="revoke-keys"
Nov 03 22:38:46.574440 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.574419    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.590729 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.590700    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbndp\" (UniqueName: \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.590847 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.590740    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.590975 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.590959    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.629289 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.629260    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"]
Nov 03 22:38:46.672495 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.672467    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:46.672616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.672602    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:46.691656 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.691603    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbndp\" (UniqueName: \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.691787 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.691658    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.691787 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.691730    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.692301 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.692281    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.694051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.694030    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.749377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.749353    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbndp\" (UniqueName: \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\") pod \"maas-api-revoke-sub-3ecd2245-kpb5g\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:46.887797 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:46.887738    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:47.046125 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:47.046075    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"]
Nov 03 22:38:47.048749 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:47.048720    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb908d3-9zq2z"]
Nov 03 22:38:47.256595 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:47.256570    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"]
Nov 03 22:38:47.258442 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:47.258418    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a6e821_56e5_4ac6_a118_4d547a10422d.slice/crio-e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f WatchSource:0}: Error finding container e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f: Status 404 returned error can't find the container with id e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f
Nov 03 22:38:48.060879 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:48.060842    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g" event={"ID":"c7a6e821-56e5-4ac6-a118-4d547a10422d","Type":"ContainerStarted","Data":"4fcc6cec1da585cbbdc0f0ca2ff1aca91f64cd34fed55af16a7ca086d5b93040"}
Nov 03 22:38:48.061315 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:48.060898    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g" event={"ID":"c7a6e821-56e5-4ac6-a118-4d547a10422d","Type":"ContainerStarted","Data":"e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f"}
Nov 03 22:38:48.172341 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:48.172296    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g" podStartSLOduration=2.17228128 podStartE2EDuration="2.17228128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:48.170879973 +0000 UTC m=+6629.802193451" watchObservedRunningTime="2026-11-03 22:38:48.17228128 +0000 UTC m=+6629.803594756"
Nov 03 22:38:48.498943 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:48.498905    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf4db16-5bb2-4a8b-9225-921f7dba147b" path="/var/lib/kubelet/pods/3bf4db16-5bb2-4a8b-9225-921f7dba147b/volumes"
Nov 03 22:38:49.071109 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:49.071079    5533 generic.go:356] "Generic (PLEG): container finished" podID="c7a6e821-56e5-4ac6-a118-4d547a10422d" containerID="4fcc6cec1da585cbbdc0f0ca2ff1aca91f64cd34fed55af16a7ca086d5b93040" exitCode=0
Nov 03 22:38:49.071616 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:49.071178    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g" event={"ID":"c7a6e821-56e5-4ac6-a118-4d547a10422d","Type":"ContainerDied","Data":"4fcc6cec1da585cbbdc0f0ca2ff1aca91f64cd34fed55af16a7ca086d5b93040"}
Nov 03 22:38:49.267055 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:38:49.267005    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:38:50.594120 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.593798    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:50.660660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.659876    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\") pod \"c7a6e821-56e5-4ac6-a118-4d547a10422d\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") "
Nov 03 22:38:50.660660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.659968    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\") pod \"c7a6e821-56e5-4ac6-a118-4d547a10422d\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") "
Nov 03 22:38:50.660660 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.660052    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\" (UniqueName: \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\") pod \"c7a6e821-56e5-4ac6-a118-4d547a10422d\" (UID: \"c7a6e821-56e5-4ac6-a118-4d547a10422d\") "
Nov 03 22:38:50.661586 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.661561    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca" pod "c7a6e821-56e5-4ac6-a118-4d547a10422d" (UID: "c7a6e821-56e5-4ac6-a118-4d547a10422d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:38:50.685370 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.685336    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp" pod "c7a6e821-56e5-4ac6-a118-4d547a10422d" (UID: "c7a6e821-56e5-4ac6-a118-4d547a10422d"). InnerVolumeSpecName "kube-api-access-wbndp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:38:50.700021 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.699999    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca" pod "c7a6e821-56e5-4ac6-a118-4d547a10422d" (UID: "c7a6e821-56e5-4ac6-a118-4d547a10422d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:38:50.775267 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.775190    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:50.775457 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.775438    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a6e821-56e5-4ac6-a118-4d547a10422d-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:50.775578 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:50.775562    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbndp\" (UniqueName: \"kubernetes.io/projected/c7a6e821-56e5-4ac6-a118-4d547a10422d-kube-api-access-wbndp\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:38:51.120058 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:51.119968    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g" event={"ID":"c7a6e821-56e5-4ac6-a118-4d547a10422d","Type":"ContainerDied","Data":"e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f"}
Nov 03 22:38:51.120343 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:51.120310    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34f6343309134ac310edc0036db0caa5121b090a2f9c5f5d20fb02c528e027f"
Nov 03 22:38:51.120476 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:51.120259    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ecd2245-kpb5g"
Nov 03 22:38:57.384465 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.384431    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"]
Nov 03 22:38:57.384940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.384846    5533 cpu_manager.go:414] "containerMap: removing container" podUID="c7a6e821-56e5-4ac6-a118-4d547a10422d" containerName="revoke-keys"
Nov 03 22:38:57.384940 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.384863    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a6e821-56e5-4ac6-a118-4d547a10422d" containerName="revoke-keys"
Nov 03 22:38:57.385059 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.384950    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a6e821-56e5-4ac6-a118-4d547a10422d" containerName="revoke-keys"
Nov 03 22:38:57.388208 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.388189    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.443070 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.443032    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"]
Nov 03 22:38:57.481234 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.481202    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qb2\" (UniqueName: \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.481390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.481254    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.481390 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.481328    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.486716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.486696    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:38:57.486789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.486736    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlx9r\""
Nov 03 22:38:57.582865 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.582839    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qb2\" (UniqueName: \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.583024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.582883    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.583024 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.582904    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.583515 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.583487    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.585173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.585153    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.640693 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.640622    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qb2\" (UniqueName: \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\") pod \"maas-api-revoke-sub-9d5a4389-ffs5s\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:57.701073 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:57.701048    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:38:58.068950 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:58.068925    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"]
Nov 03 22:38:58.071029 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:38:58.070988    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdee5dda_2a20_4145_8f10_34c096d8a9de.slice/crio-659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982 WatchSource:0}: Error finding container 659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982: Status 404 returned error can't find the container with id 659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982
Nov 03 22:38:58.162883 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:58.162854    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s" event={"ID":"cdee5dda-2a20-4145-8f10-34c096d8a9de","Type":"ContainerStarted","Data":"659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982"}
Nov 03 22:38:59.170207 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:59.170165    5533 generic.go:356] "Generic (PLEG): container finished" podID="cdee5dda-2a20-4145-8f10-34c096d8a9de" containerID="5ef5ab3b67f7777b653c4bc6ca15a3dd8757437c94e911570779d38965e61544" exitCode=0
Nov 03 22:38:59.170566 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:38:59.170248    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s" event={"ID":"cdee5dda-2a20-4145-8f10-34c096d8a9de","Type":"ContainerDied","Data":"5ef5ab3b67f7777b653c4bc6ca15a3dd8757437c94e911570779d38965e61544"}
Nov 03 22:39:00.622114 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.621822    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:39:00.733051 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.733019    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\") pod \"cdee5dda-2a20-4145-8f10-34c096d8a9de\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") "
Nov 03 22:39:00.733236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.733088    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\" (UniqueName: \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\") pod \"cdee5dda-2a20-4145-8f10-34c096d8a9de\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") "
Nov 03 22:39:00.733236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.733123    5533 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\") pod \"cdee5dda-2a20-4145-8f10-34c096d8a9de\" (UID: \"cdee5dda-2a20-4145-8f10-34c096d8a9de\") "
Nov 03 22:39:00.733422 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.733394    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca" pod "cdee5dda-2a20-4145-8f10-34c096d8a9de" (UID: "cdee5dda-2a20-4145-8f10-34c096d8a9de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:39:00.735587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.735555    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2" pod "cdee5dda-2a20-4145-8f10-34c096d8a9de" (UID: "cdee5dda-2a20-4145-8f10-34c096d8a9de"). InnerVolumeSpecName "kube-api-access-79qb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:39:00.735833 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.735808    5533 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca" pod "cdee5dda-2a20-4145-8f10-34c096d8a9de" (UID: "cdee5dda-2a20-4145-8f10-34c096d8a9de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:39:00.834262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.834198    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-xks-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:39:00.834421 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.834408    5533 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdee5dda-2a20-4145-8f10-34c096d8a9de-maas-api-openshift-ca\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:39:00.834492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:00.834483    5533 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qb2\" (UniqueName: \"kubernetes.io/projected/cdee5dda-2a20-4145-8f10-34c096d8a9de-kube-api-access-79qb2\") on node \"ip-10-0-140-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 22:39:01.288436 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:01.288394    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s" event={"ID":"cdee5dda-2a20-4145-8f10-34c096d8a9de","Type":"ContainerDied","Data":"659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982"}
Nov 03 22:39:01.293803 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:01.293781    5533 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="659bc681205892dce70602412a3c829c642551312fd0012287f088f6d43f1982"
Nov 03 22:39:01.293977 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:01.288872    5533 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5a4389-ffs5s"
Nov 03 22:39:10.105702 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:10.105665    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kcbgn"]
Nov 03 22:39:19.259420 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:39:19.259383    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:39:20.047509 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:20.047474    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"]
Nov 03 22:39:20.094682 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:20.094628    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a08b8c9-lrqsq"]
Nov 03 22:39:20.491667 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:20.491624    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd56813-91fb-453a-8cf9-4f8b41c20b1d" path="/var/lib/kubelet/pods/efd56813-91fb-453a-8cf9-4f8b41c20b1d/volumes"
Nov 03 22:39:23.305889 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:23.305857    5533 scope.go:122] "RemoveContainer" containerID="92c0f5994483d4f5e5cd303fbf8dbaa6d3effb6bd4312c63199ea44c09fbadff"
Nov 03 22:39:23.315003 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:23.314977    5533 scope.go:122] "RemoveContainer" containerID="255c2eed80a6c7471017c65b81bd407c67aaf5f5427a9bd84605813f65c670fc"
Nov 03 22:39:37.718643 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:37.718613    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/1.log"
Nov 03 22:39:39.478311 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:39.478282    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/0.log"
Nov 03 22:39:40.863279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:40.863248    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-067b78fe-rzgcq_1a4c1f97-2c9b-4750-91e5-a20653fb7fe5/revoke-keys/0.log"
Nov 03 22:39:41.091824 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:41.091795    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-10ff5642-mpbfq_b9f4415e-e795-474e-9563-d32133b00ffa/revoke-keys/0.log"
Nov 03 22:39:41.323055 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:41.323030    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3ecd2245-kpb5g_c7a6e821-56e5-4ac6-a118-4d547a10422d/revoke-keys/0.log"
Nov 03 22:39:41.537605 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:41.537582    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-46064687-hjn2c_b83154ca-e854-44c2-b366-79f1e62023ab/revoke-keys/0.log"
Nov 03 22:39:41.762761 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:41.762732    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47430f29-5bcch_5cdccf42-4fce-4b7e-a63d-e7060b2b0863/revoke-keys/0.log"
Nov 03 22:39:42.002455 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:42.002433    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-76a5d921-jcmd8_0858251b-00c9-4d10-a085-535a84feee15/revoke-keys/0.log"
Nov 03 22:39:42.220783 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:42.220756    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7b44d61d-5gmmx_e00b4e74-19c8-4bf4-b0a5-c9e9a7dfea6e/revoke-keys/0.log"
Nov 03 22:39:42.439710 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:42.439683    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8a300810-pdmf7_baa67633-72d3-49ca-92f1-197777534336/revoke-keys/0.log"
Nov 03 22:39:42.658173 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:42.658106    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-909045e6-rzmcg_b2e7808a-fee4-4839-89d3-21574b6185a1/revoke-keys/0.log"
Nov 03 22:39:42.880427 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:42.880403    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-95b538da-tvgxd_a6811be6-7532-4fff-86a2-ee0ef5d94c98/revoke-keys/0.log"
Nov 03 22:39:43.090169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:43.090143    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9a476202-f6khf_1a4bf38e-66dc-4a27-bcbe-dcd84c9850a2/revoke-keys/0.log"
Nov 03 22:39:43.294185 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:43.294163    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9d5a4389-ffs5s_cdee5dda-2a20-4145-8f10-34c096d8a9de/revoke-keys/0.log"
Nov 03 22:39:43.519758 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:43.519735    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b2a22b3e-dm8dr_73eeab38-1902-4c50-9763-2bc0ba9c9829/revoke-keys/0.log"
Nov 03 22:39:43.747082 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:43.747051    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b37834e6-hp2q5_be79f06a-aedc-4093-9e1c-a78d01d0bd54/revoke-keys/0.log"
Nov 03 22:39:43.969959 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:43.969933    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-efd89c07-g8kzj_9e6f6fa8-5410-4878-a500-063ee99e9703/revoke-keys/0.log"
Nov 03 22:39:44.189814 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:44.189789    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f39f0c36-ng989_86949266-8b11-4d28-ab2b-0c1e1ea6ccde/revoke-keys/0.log"
Nov 03 22:39:44.420581 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:44.420557    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6ndhn_078be369-f4bf-4802-ab25-05a5c196b0d7/postgres/0.log"
Nov 03 22:39:47.194194 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:47.194168    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csxht_8fac1941-f4d4-4e7d-ad78-a6c93434daf3/kuadrant-console-plugin/0.log"
Nov 03 22:39:47.435589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:47.435553    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2drq5_94c0ddb3-f9d6-42db-83d6-304fc86bc632/manager/0.log"
Nov 03 22:39:48.006190 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:48.006161    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kcbgn_18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3/limitador/0.log"
Nov 03 22:39:48.244653 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:48.244607    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ngx24_128dab95-58a5-4fc4-bc64-21b427628fc2/manager/0.log"
Nov 03 22:39:49.261323 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:39:49.261287    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:39:49.269978 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:49.269951    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc_08564a25-5b1d-419c-b108-18b2c11d7f32/istio-proxy/0.log"
Nov 03 22:39:50.602577 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:50.602546    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-5swkz_25810e8f-c406-4e5b-9194-64d849a8c95e/istio-proxy/0.log"
Nov 03 22:39:50.933858 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:50.933828    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-7lrp6_6d70fc5c-1dba-4651-b45b-299fa16b1692/istio-proxy/0.log"
Nov 03 22:39:52.719137 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:52.719108    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-z2qhq_fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67/istio-proxy/0.log"
Nov 03 22:39:53.264754 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:53.264722    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-857c5f8fc8-tkszh_273ee43f-335f-4f07-8949-d3e33b4c55bc/payload-pre-processing/0.log"
Nov 03 22:39:53.480382 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:53.480338    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-679fff6fd4-krq72_7c866d43-d902-448b-b17b-89b9ef7bdb4f/payload-processing/0.log"
Nov 03 22:39:55.153027 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:55.152986    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-89cxr_a9a714ca-1b70-4236-ab4f-022c4c3449ef/main/0.log"
Nov 03 22:39:55.720262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:55.720236    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-5vdsp_7cdac0ca-f0ca-4565-93cb-b8b8a42b6c21/main/0.log"
Nov 03 22:39:56.278279 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:56.278245    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-vfz7m_b3d34006-15de-452d-a377-c7bd2f14bdeb/main/0.log"
Nov 03 22:39:56.827339 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:39:56.827305    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-p8jgq_3b4da839-0fd8-4d9e-81a2-371496377bba/main/0.log"
Nov 03 22:40:03.046169 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:03.046131    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"]
Nov 03 22:40:03.092932 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:03.092901    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46064687-hjn2c"]
Nov 03 22:40:04.492698 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:04.492660    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83154ca-e854-44c2-b366-79f1e62023ab" path="/var/lib/kubelet/pods/b83154ca-e854-44c2-b366-79f1e62023ab/volumes"
Nov 03 22:40:06.793665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.789902    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vw5md/must-gather-zqdv9"]
Nov 03 22:40:06.793665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.790761    5533 cpu_manager.go:414] "containerMap: removing container" podUID="cdee5dda-2a20-4145-8f10-34c096d8a9de" containerName="revoke-keys"
Nov 03 22:40:06.793665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.790781    5533 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdee5dda-2a20-4145-8f10-34c096d8a9de" containerName="revoke-keys"
Nov 03 22:40:06.793665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.790963    5533 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdee5dda-2a20-4145-8f10-34c096d8a9de" containerName="revoke-keys"
Nov 03 22:40:06.795990 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.795964    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:06.871672 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.871642    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a95522b-2890-4f54-91e4-63eaff9b7893-must-gather-output\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:06.871811 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.871763    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4lt\" (UniqueName: \"kubernetes.io/projected/0a95522b-2890-4f54-91e4-63eaff9b7893-kube-api-access-bl4lt\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:06.892807 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.892784    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vw5md\"/\"default-dockercfg-s87vf\""
Nov 03 22:40:06.893113 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.893085    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vw5md\"/\"openshift-service-ca.crt\""
Nov 03 22:40:06.893230 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.893122    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vw5md\"/\"kube-root-ca.crt\""
Nov 03 22:40:06.895892 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.895869    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vw5md/must-gather-zqdv9"]
Nov 03 22:40:06.972874 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.972844    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4lt\" (UniqueName: \"kubernetes.io/projected/0a95522b-2890-4f54-91e4-63eaff9b7893-kube-api-access-bl4lt\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:06.973006 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.972902    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a95522b-2890-4f54-91e4-63eaff9b7893-must-gather-output\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:06.973262 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:06.973244    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a95522b-2890-4f54-91e4-63eaff9b7893-must-gather-output\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:07.030214 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:07.030191    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4lt\" (UniqueName: \"kubernetes.io/projected/0a95522b-2890-4f54-91e4-63eaff9b7893-kube-api-access-bl4lt\") pod \"must-gather-zqdv9\" (UID: \"0a95522b-2890-4f54-91e4-63eaff9b7893\") " pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:07.110157 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:07.110107    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vw5md/must-gather-zqdv9"
Nov 03 22:40:07.477119 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:07.477074    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vw5md/must-gather-zqdv9"]
Nov 03 22:40:07.478375 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:40:07.478348    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a95522b_2890_4f54_91e4_63eaff9b7893.slice/crio-72ba58ddf303375a650f0b102e335f22915c3dc40e3bf320ad25941da4eb69e7 WatchSource:0}: Error finding container 72ba58ddf303375a650f0b102e335f22915c3dc40e3bf320ad25941da4eb69e7: Status 404 returned error can't find the container with id 72ba58ddf303375a650f0b102e335f22915c3dc40e3bf320ad25941da4eb69e7
Nov 03 22:40:07.480252 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:07.480234    5533 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:40:07.663913 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:07.663881    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vw5md/must-gather-zqdv9" event={"ID":"0a95522b-2890-4f54-91e4-63eaff9b7893","Type":"ContainerStarted","Data":"72ba58ddf303375a650f0b102e335f22915c3dc40e3bf320ad25941da4eb69e7"}
Nov 03 22:40:14.048064 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.048017    5533 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"]
Nov 03 22:40:14.093665 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.093628    5533 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909045e6-rzmcg"]
Nov 03 22:40:14.495591 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.495562    5533 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e7808a-fee4-4839-89d3-21574b6185a1" path="/var/lib/kubelet/pods/b2e7808a-fee4-4839-89d3-21574b6185a1/volumes"
Nov 03 22:40:14.717574 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.717541    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vw5md/must-gather-zqdv9" event={"ID":"0a95522b-2890-4f54-91e4-63eaff9b7893","Type":"ContainerStarted","Data":"22b79bfe0c0e8be92cc5363d475965079d0ed15bc1a05c4a00a1f37e7535096d"}
Nov 03 22:40:14.717779 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.717583    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vw5md/must-gather-zqdv9" event={"ID":"0a95522b-2890-4f54-91e4-63eaff9b7893","Type":"ContainerStarted","Data":"a7d9d02d8761531492e2badb2e6b3e0a5aa1c90c93389e515c4964a15503ddfc"}
Nov 03 22:40:14.828676 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:14.828602    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vw5md/must-gather-zqdv9" podStartSLOduration=1.990000494 podStartE2EDuration="8.828588451s" totalImagesPullingTime="6.838587957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:14.82677988 +0000 UTC m=+6716.458093359" watchObservedRunningTime="2026-11-03 22:40:14.828588451 +0000 UTC m=+6716.459901928"
Nov 03 22:40:17.393236 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:17.393200    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-stvfh_c63d20b7-d348-4c0b-9a07-2e2b45ccd0ec/global-pull-secret-syncer/0.log"
Nov 03 22:40:19.260939 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:40:19.260900    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:40:19.803878 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:19.803847    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/1.log"
Nov 03 22:40:19.814589 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:19.814559    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/0.log"
Nov 03 22:40:20.289575 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:20.289542    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qj2ct_33fd3340-3104-450d-a434-0e89d73d7331/konnectivity-agent/0.log"
Nov 03 22:40:21.558715 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:21.558686    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-144.us-west-2.compute.internal_8b940c9766e72993f1a0cb04738b1af3/haproxy/0.log"
Nov 03 22:40:23.342561 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:23.342525    5533 scope.go:122] "RemoveContainer" containerID="92ef4095272e9928a3b7d931524f610946e2e0e47a61693decbefff02993bcf8"
Nov 03 22:40:23.352124 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:23.352100    5533 scope.go:122] "RemoveContainer" containerID="453c12142551dfcace36e577a3ebece8e6de4a5ec0cc1d838e158e38c25847da"
Nov 03 22:40:23.360376 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:23.360358    5533 scope.go:122] "RemoveContainer" containerID="88c3f93153d58c43983a5f2e9cc8c6dca1dbc40fd63e021604838c0bd860c252"
Nov 03 22:40:25.984477 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:25.984443    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/1.log"
Nov 03 22:40:26.000461 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.000434    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/0.log"
Nov 03 22:40:26.760430 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.760388    5533 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"]
Nov 03 22:40:26.769624 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.769602    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.818773 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.818745    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"kube-root-ca.crt\""
Nov 03 22:40:26.846399 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.846374    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7cj\" (UniqueName: \"kubernetes.io/projected/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-kube-api-access-pz7cj\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.846540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.846409    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-sys\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.846540 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.846508    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-lib-modules\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.846646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.846566    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-podres\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.846646 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.846593    5533 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-proc\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.865434 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.865406    5533 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd72m\"/\"openshift-service-ca.crt\""
Nov 03 22:40:26.865780 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.865701    5533 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cd72m\"/\"default-dockercfg-77mtt\""
Nov 03 22:40:26.868575 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.868553    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"]
Nov 03 22:40:26.947719 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947694    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-podres\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.947844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947735    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-proc\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.947844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947805    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7cj\" (UniqueName: \"kubernetes.io/projected/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-kube-api-access-pz7cj\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.947844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947810    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-podres\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.947844 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947839    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-sys\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.948069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947866    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-proc\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.948069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947886    5533 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-lib-modules\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.948069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.947930    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-sys\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:26.948069 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:26.948004    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-lib-modules\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:27.005741 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.005705    5533 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7cj\" (UniqueName: \"kubernetes.io/projected/4c80c13f-c3fe-463d-a3cd-ed814060f2b0-kube-api-access-pz7cj\") pod \"perf-node-gather-daemonset-8bpkt\" (UID: \"4c80c13f-c3fe-463d-a3cd-ed814060f2b0\") " pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:27.081828 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.081766    5533 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:27.249359 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.249335    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csxht_8fac1941-f4d4-4e7d-ad78-a6c93434daf3/kuadrant-console-plugin/0.log"
Nov 03 22:40:27.450401 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.450365    5533 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"]
Nov 03 22:40:27.451622 ip-10-0-140-144 kubenswrapper[5533]: W1103 22:40:27.451596    5533 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c80c13f_c3fe_463d_a3cd_ed814060f2b0.slice/crio-b3be4bec93185736e6dce06febb63df693039cf6e645dbe3c05dfc530a9e65e4 WatchSource:0}: Error finding container b3be4bec93185736e6dce06febb63df693039cf6e645dbe3c05dfc530a9e65e4: Status 404 returned error can't find the container with id b3be4bec93185736e6dce06febb63df693039cf6e645dbe3c05dfc530a9e65e4
Nov 03 22:40:27.717160 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.717088    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2drq5_94c0ddb3-f9d6-42db-83d6-304fc86bc632/manager/0.log"
Nov 03 22:40:27.794716 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.794680    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt" event={"ID":"4c80c13f-c3fe-463d-a3cd-ed814060f2b0","Type":"ContainerStarted","Data":"5903a84839a6ba44f1812b50e8d0b0717e0b60239719320ea4f3f3e588f61544"}
Nov 03 22:40:27.794895 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.794778    5533 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt" event={"ID":"4c80c13f-c3fe-463d-a3cd-ed814060f2b0","Type":"ContainerStarted","Data":"b3be4bec93185736e6dce06febb63df693039cf6e645dbe3c05dfc530a9e65e4"}
Nov 03 22:40:27.794895 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.794810    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:27.906086 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:27.906038    5533 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt" podStartSLOduration=1.90602431 podStartE2EDuration="1.90602431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:27.903599762 +0000 UTC m=+6729.534913323" watchObservedRunningTime="2026-11-03 22:40:27.90602431 +0000 UTC m=+6729.537337813"
Nov 03 22:40:28.114280 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:28.114207    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kcbgn_18eb3d6d-1d38-4c34-a7f2-46092ddb2ad3/limitador/0.log"
Nov 03 22:40:28.667647 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:28.667613    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ngx24_128dab95-58a5-4fc4-bc64-21b427628fc2/manager/0.log"
Nov 03 22:40:32.360771 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:32.360740    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/1.log"
Nov 03 22:40:32.386099 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:32.386068    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77b845648-rzlw2_7e9c8826-0e09-4d08-836c-186eab92e73e/manager/0.log"
Nov 03 22:40:33.811180 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:33.811146    5533 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cd72m/perf-node-gather-daemonset-8bpkt"
Nov 03 22:40:36.893599 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:36.893569    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ssfh_b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/node-exporter/0.log"
Nov 03 22:40:37.320587 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:37.320562    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ssfh_b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/kube-rbac-proxy/0.log"
Nov 03 22:40:37.504789 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:37.504757    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fc46679f7r2ktc_08564a25-5b1d-419c-b108-18b2c11d7f32/istio-proxy/0.log"
Nov 03 22:40:37.769344 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:37.769319    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ssfh_b8dfb03f-f653-43e7-9a8a-4a61bbb2220a/init-textfile/0.log"
Nov 03 22:40:42.142764 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:42.142733    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/prometheus/0.log"
Nov 03 22:40:42.325211 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:42.325171    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-5swkz_25810e8f-c406-4e5b-9194-64d849a8c95e/istio-proxy/0.log"
Nov 03 22:40:42.566859 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:42.566827    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-7lrp6_6d70fc5c-1dba-4651-b45b-299fa16b1692/istio-proxy/0.log"
Nov 03 22:40:42.588216 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:42.588187    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/config-reloader/0.log"
Nov 03 22:40:43.033941 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:43.033909    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/thanos-sidecar/0.log"
Nov 03 22:40:43.462377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:43.462342    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/kube-rbac-proxy-web/0.log"
Nov 03 22:40:43.891129 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:43.891052    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/kube-rbac-proxy/0.log"
Nov 03 22:40:44.055701 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:44.055667    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-z2qhq_fea9c0e1-cbc2-4b28-a4e0-ca51f3ed6a67/istio-proxy/0.log"
Nov 03 22:40:44.326377 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:44.326344    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/kube-rbac-proxy-thanos/0.log"
Nov 03 22:40:44.467492 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:44.467465    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-857c5f8fc8-tkszh_273ee43f-335f-4f07-8949-d3e33b4c55bc/payload-pre-processing/0.log"
Nov 03 22:40:44.674486 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:44.674463    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-679fff6fd4-krq72_7c866d43-d902-448b-b17b-89b9ef7bdb4f/payload-processing/0.log"
Nov 03 22:40:44.767094 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:44.767061    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_225bfb37-e79e-4293-ad0f-befcfc42f1ae/init-config-reloader/0.log"
Nov 03 22:40:45.208712 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:45.208681    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qxlz9_37651c6c-a9ce-4c30-af0f-7e74869a7ef4/prometheus-operator/0.log"
Nov 03 22:40:45.626110 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:45.626025    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qxlz9_37651c6c-a9ce-4c30-af0f-7e74869a7ef4/kube-rbac-proxy/0.log"
Nov 03 22:40:49.260710 ip-10-0-140-144 kubenswrapper[5533]: E1103 22:40:49.260666    5533 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mv9sd" containerName="extractor"
Nov 03 22:40:50.521558 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:50.521531    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-csxht_8fac1941-f4d4-4e7d-ad78-a6c93434daf3/kuadrant-console-plugin/0.log"
Nov 03 22:40:50.773159 ip-10-0-140-144 kubenswrapper[5533]: I1103 22:40:50.773072    5533 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2drq5_94c0ddb3-f9d6-42db-83d6-304fc86bc632/manager/0.log"